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ABSTRACT 


The  purpose  of  this  thesis  is  to  explore  and  identify  key  problems  in  Developmental 
Testing  and  Evaluation  (DT&E)  in  the  United  States  Army.  A  comparative  analysis  of 
several  programs  is  conducted  to  determine  common  developmental  testing  problems. 
These  problems  are  analyzed  and  a  set  of  conclusions  and  recommendations  for  future 
testing  is  developed.  The  thesis  provides  the  reader  Nvith  a  current  understanding  of  the 
Department  of  Defense  (DOD)  test  structure,  its  relationship  to  the  acquisition  process,  and 
the  Department  of  the  Army  test  agencies  involved  in  Developmental  Test  and  Evaluation. 
The  agencies  identified  five  common  categories  of  problems  across  seven  systems.  These 
systems  included:  the  Abrams  Main  Battle  Tank  Block  II  upgrade  (Ml  A2);  the  Javelin,  a 
man  portable  antitank  weapon,  (AAWS-M);  Enhanced  Position  Location  Reporting  System 
(EPLRS);  the  Avenger,  a  mounted  Air  Defense  system,  the  Kiowa  Warrior,  an  armed  scout 
helicopter;  the  Maneuver  Control  System  (MCS),  a  command  and  control  system;  and  the 
Family  of  Medium  Tactical  Vehicles  (FMTV).  The  thesis  concludes  by  providing 
recommendations  to  help  future  testers,  evaluators  and  program  managers  to  better  prepare 
for  DT&E  in  the  acquisition  life  cycle. 
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I.  INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  explore  and  identify  key  problems  in  Developmental 
Testing  and  Evaluation  (DT&E)  in  the  United  States  Army.  A  comparative  analysis  of 
several  programs  is  conducted  to  determine  common  developmental  testing  problems. 
These  problems  are  analyzed  and  a  set  of  conclu.sions  and  recommendations  for  future 
testing  is  developed.  The  thesis  provides  tfje  reader  with  a  current  understanding  of  the 
Department  of  Defense  (DOD)  test  structure,  its  relationship  to  the  acquisition  process,  and 
the  Department  of  the  Army  test  agencies  involved  in  Developmental  Test  and  Evaluation. 
The  thesis  concludes  by  providing  recommendations  to  help  future  testers,  evaluators  and 
program  managers  to  better  prepare  for  DT&E  in  the  acquisition  life  cycle. 

B.  BACKGROUND 

Test  and  Evaluation  (T&E)  is  required  in  the  acquisition  life  of  any  Department  of 
Defense  system.  Developmental  testing  is  critical  to  a  program  moving  on  in  the 
acquisition  process.  Developmental  testing,  like  the  entire  acquisition  process  is  subject  to 
budget  cycles,  agency  conflicts,  turf  battles,  congressional  influence  and  other  political 
factors  all  of  which  can  cause  problems.  Problems  in  developmental  testing  have  caused 
schedule  delays,  inadequate  or  useless  data  and  unplanned  expenditures  of  large  amounts 
of  money  .and  other  resources  to  resolve  the  situations.  In  the  current  environment  of 
declining  budgets,  such  problems  can  result  in  program  slowdown  or  even  cancellation. 

There  are  two  major  categories  of  test  and  evaluation;  (1)  Developmental  Test  and 
Evaluation  (DT&E)  and  (2)  Operational  Test  and  Evaluation  (OT&E).  Operational  Testing 
as  well  as  other  types  of  testing  are  described,  but  the  focus  of  this  thesis  is  DT&E. 


Developmental  Test  and  Evaluation  (DT&E)  is  conducted  throughout  the  acquisition 
process  to  ensure  the  acquisition  and  fielding  of  an  effective  and  supportable  system.  Early 
DT&E  is  normally  conducted  by  the  sub-contractors  and  the  prime  contractor.  The  sub¬ 
contractors  test  the  components  as  they  ve  developed  and  the  prime  is  interested  in  testing 
the  total  system  as  the  components  are  integrated.  The  Government  test  agencies  are  more 
involved  during  the  later  acquisition  phases  to  demonstrate  how  well  the  weapon  system 
meets  its  technical  requirements. 

There  are  several  key  players  involved  in  DT&E  in  the  U.  S.  Army,  DT&E  is 
normally  planned,  coordinated,  conducted  and  monitored  by  the  United  States  Army  Test 
and  Evaluation  Command  (TECOM),  a  sub  command  of  the  Army  Materiel  Command 
(AMC).  The  test  design,  evaluation  and  analysis  role  of  DT&E  is  conducted  by  the  United 
States  Army  Materiel  Systems  Analysis  Activity  (AMSAA).  Other  critical  players  include 
the  future  users  or  Combat  Developers  from  the  United  States  Army  Training  and  Doctrine 
Command,  (TRADOC),  and  the  Program  Management  Office  (PMO)  charged  with  the 
overall  acquisition  of  the  system  including  testing. 

C.  OBJECTIVES  OF  THE  THESIS 

The  objective  of  the  thesis  is  to  explore  and  identify  common  problems  of 
developmental  testing  in  the  United  States  Army  from  the  perspective  of  the  different 
agencies  involved.  The  agencies  or  groups  focused  on  were:  the  Program  Management 
Office  (PMO),  the  testers  and  their  facilities,  the  analysts/evaluators,  the  contractors,  and 
the  users. 

The  Program  Management  Office  (PMO)  is  ultimately  responsible  for  all  aspects  of 
any  program  including  testing.  The  PMO  normally  maintains  a  section  whose  functions 
include  test  coordination.  The  Army's  Test  and  Evaluation  Command  (TECOM)  provides 
most  of  the  resources  --  testers  and  facilities  -  for  DT&E  in  the  Army  today.  Analysis  and 
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evaluation  for  Army  DT&E  is  normally  conducted  by  the  Army  Materiel  Systems  Analysis 
Agency  (AMSAA),  This  agency  impacts  the  analytical  and  statistical  side  of  the  DT&E. 
The  contractor  or  builder  plays  a  signiflcant  role  in  the  test,  fix,  retest  scenario  of  the  early 
stages  of  a  weapon  system  development  Fmally,  the  users  provide  the  need  and  request 
the  capabilities  that  the  system  under  test  will  hopefully  provide.  The  user  nr  Combat 
Developer  is  usually  represented  by  a  TRADOC  Systems  Manager  (TSM)  or  by  a 
combined  TRADOC  organization  such  as  the  Combined  Arms  Center  at  Fort  Leavenworth, 
Kansas. 

The  organization  and  basic  DOD  test  process  is  described  followed  by  the  Army  test 
structure  down  to  the  agencies  and  offices  that  are  the  focus  of  the  data  collection.  Data  are 
presented  and  once  the  developmental  test  problems  are  identified,  they  are  analyzed  and  a 
set  of  recommendations  ^s  developed  from  this  information.  The  recommendations  can 
then  be  used  by  program  managers,  testers  and  evaluators  to  help  them  assess  and  prepare 
their  system  for  developmental  testing. 

D.  RESEARCH  QUESTIONS 

The  primary  research  question  for  this  thesis  is: 

What  is  die  most  significant  problem  in  conducting  developmental  testing? 

The  following  are  subsidiary  research  questions  to  help  develop  and  defuie  the  primary 
research  question. 

1 .  What  are  the  common  developmental  test  problem  areas  across  agencies? 

2 .  What  are  the  common  developmental  test  problem  areas  across  types  of  systems  or 

acquisition  strategies? 

3 .  To  what  extent  do  these  problems  endanger  program  success? 

4.  To  what  extent  do  diese  proolems  impact  cost  and  schedule? 

5.  What  can  be  done  by  program  managers,  testers  and  evaluators  or  others  to 

improve  the  preparation  and  conduct  of  developmental  testing? 
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E.  SCOPE  AND  LIMn  ATIONS 

The  scope  of  the  research  is  limited  to  Acquisition  Category  (AC  AT)  I  and  II  systems 
tested  at  United  States  Army  Test  and  Evaluation  Command  (TECOM)  facilities  within  the 
United  States,  Seven  programs  under  test  or  tested  in  the  past  10  years  are  researched  and 
analyzed.  The  systems  include  the  Abrams  Main  Battle  Tank  Block  D  upgrade  (Ml  A2);  the 
Javelin,  a  man  portable  antitank  weapon,  (AAWS-M);  Enhanced  Position  Location 
Reporting  System  (EPLRS);  the  Avenger,  a  mounted  Air  Defense  system,  the  Kiowa 
Warrior,  an  armed  scout  helicopter;  the  Maneuver  Control  System  (MCS),  a  command  and 
control  system;  and  the  Family  of  Medium  Tactical  Vehicles  (FMTV).  These  program.^ 
represent  different  types  of  systems  from  electronic  communications  and  software  to  major 
weapon  systems  and  represent  different  types  of  acquisitions  from  system  upgrades  and 
Non-Developmental  Items  (NDI)  to  full  scale  development.  Other  information  will  come 
from  literature  searches  and  interviews  as  indicated  below. 

F.  RESEARCH  METHODOLOGY 

Research  investigation  included  a  literature  search  of  After  Action  Reports  from  Test 
and  Evaluation  Command  (TECOM),  lessons  learned  reports  of  major  systems.  General 
Accounting  Office  Reports,  Congressional  Subcommittee  Reports,  Developmental  Test  and 
Evaluation  Reports  and  technical  and  professional  journals.  Developmental  testing  does 
not  normally  receive  the  public  scrutiny  of  operational  testing  and  therefore  much  of  the 
information  was  gathered  through  the  interview  process.  Interviews  were  conducted  with 
program  office  personnel,  program  test  offreers,  analysis  personnel,  Combat  Developers 
and  contractors  who  participated  in  developmental  testing  of  the  major  programs  already 
mentioned.  Supervisors  from  AMSAA  and  TECOM  with  years  of  DT&E  experience  in  the 
Army  were  also  interviewed.  Interviews  were  conducted  in  person,  over  the  phone  and 
through  the  use  of  video  teleconferencing. 
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G .  ORGANIZATION  OF  THE  STUDY 

The  thesis  is  organized  into  the  following  chapters. 

Chapter  D:  Background  -  This  chapter  contains  historical  information  on  DOD  testing 
and  describes  the  current  DOD  test  structuie  within  the  acquisition  process.  The  chapter 
then  describes  the  test  structure  in  the  U.  S.  Army  focusing  on  DT&E  and  the  key  players 
in  Army  DT&E. 

Chapter  El:  Methodology  and  Data  Summary  -  This  chapter  explains  the  methods 
used  for  executing  the  research  design  and  structure  of  the  analysis.  The  chapter  then 
presents  the  data  that  were  used  for  the  analysis.  Most  of  the  data  are  in  the  form  of 
interyiews. 

Chapter  IV:  Results  and  Analysis  --  This  chapter  analyzes  the  data  and  indicates  their 
implications. 

Chapter  V:  Conclusions  and  Recommendations  —  This  chapter  contains  a  summary  of 
the  principal  findings  of  the  thesis  and  offers  recommendations  for  future  use. 


5 


II.  BACKGROUND 


A.  INTRODUCTION 

This  chapter  provides  a  basic  history  of  testing  in  the  Department  of  Defense  (DOD) 
with  emphasis  on  the  last  twenty  years.  The  chapter  describes  the  four  major  categories  of 
testing  namely  Development  Testing,  Operational  Testing,  Joint  Testing  and  Multi- Service 
Testing.  The  chapter  then  describes  testing  within  the  acquisition  process  and  concludes 
with  the  description  of  the  Developmental  Testing  function  within  the  United  States  Army. 

Test  and  Evaluation  (T&E)  is  a  critical  part  of  the  acquisition  process.  "Test"  denotes 
the  actual  testing  of  hardware  or  software  to  obtain  data.  These  data  are  valuable  in 
developing  new  capabilities,  managing  the  process,  or  making  decisions  on  the  allocation 
of  resources.  "Evaluation"  denotes  the  process  whereby  data  are  logically  assembled  and 
analyzed  to  aid  in  making  systematic  decisions.  "Test  and  Evaluation  is  the  process  by 
which  a  system  or  components  are  compared  against  requirements  and  specifications 
through  testing."  [Ref.  1] 

The  planning  and  conducting  of  T&E  exists  throughout  the  acquisition  cycle.  There  is 
the  need  for  thorough,  logical,  systematic,  and  early  test  planning  and  the  feedback  of  well 
documented,  unbiased  test  and  evaluation  results  to  system  developers,  users,  and  decision 
makers.  The  purpose  of  Test  and  Evaluation  in  a  defense  system's  development  and 
acquisition  program  is  to  identify  the  areas  of  risk  to  be  reduced  or  eliminated.  During  the 
early  phases  of  development,  T&E  is  conducted  to  demonstrate  the  feasibility  of  conceptual 
approaches,  to  minimize  design  risk,  to  identify  design  alternatives,  to  compare  and  analyze 
tradeoffs,  and  to  estimate  operational  effectiveness  and  suitability.  [Ref.  2] 
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B.  HISTORY 

1 .  World  War  U  to  1960's 

Equipment  testing  has  been  a  part  of  the  procurement  process  throughout  the 
nation’s  history.  For  decades  testing  remained  informal,  generally  "ad  hoc"  and  evolved 
along  with  the  procurement  process.  During  World  War  n  the  procurement  process  began 
to  take  on  a  more  formalized  management  approach.  As  the  procurement  process  evolved 
so  did  testing.  Testing  was  conducted  throughout  World  War  II.  There  were  engineering 
tests  to  test  engineering  and  scientific  characteristics  and  Service  tests  conducted  by  the 
various  branches  to  determine  if  the  equipment  was  sufficient  for  field  use.  Testing  was 
basically  sequential,  lengthy  and  dependent  on  the  need  of  the  equipment. 

The  war  ended  with  much  equipment  still  in  testing  and  a  recognition  that 
research,  development  and  evaluation  must  continue  to  be  a  peacetime  effort.  Those 
involved  in  testing  during  the  war  determined  that  testing  could  be  greatly  simplified  under 
more  completely  integrated  development  agencies.  The  development  and  continued  testing 
of  the  atomic  bomb  offered  a  ready  example  of  such  an  integrated  and  expedited  effort.  An 
R&D  Division  was  developed  in  the  Army  General  Staff  in  1946,  however  it  soon  fell 
victim  to  demobilization  and  by  1948  the  functions  of  the  R&D  division  were  assigned  to  a 
subgroup  in  the  Logistics  Division.  [Ref.  3] 

Following  the  war  most  of  the  research,  development  and  testing  conducted  in 
DOD  dealt  with  rockets  and  missiles.  During  this  time  the  procurement  process  was 
evolving  and  terminology  like  systems  engineering,  operations  research,  project  offices  and 
contracted  engineering  support  started  to  come  into  play. 

2.  1960's  and  70's 

The  1960’s  saw  the  formalization  of  the  acquisition  process  and  subsequently  the 
formalization  of  the  testing  process.  Initially  Secretary  of  Defense  McNamara  took  a 
business  approach  to  the  acquisition  process  fostering  total  package  procurement,  strong 
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centralized  civilian  control  and  concurrency.  In  the  early  1970's,  Secretary  of  Defense 
Laird  and  his  deputy,  David  Packard,  promoted  decentralization  and  a  “fly  before  buy” 
mentality.  The  Department  of  Defense  (DOD)  test  policy  within  the  acquisition  system 
became  more  formalized  and  placed  greater  emphasis  on  Test  and  Evaluation  (T&E)  as  a 
continuing  function  throughout  the  acquisition  cycle.  The  policies  stressed  the  use  of  T&E 
to  reduce  risk  and  provide  a  continuous  estimate  on  the  system's  effectiveness  and 
suitability.  To  meet  these  objectives  it  was  important  that  sqjpropriate  test  activities  be  fully 
integrated  into  the  overall  development  process.  [Ref.  4]  It  was  also  during  this  time  that 
both  the  Army  and  tne  Air  Force  were  given  a  congressional  directive  to  establish 
independent  operational  test  agencies. 

3.  1980's 

Early  in  the  1980’s  the  acquisition  system  was  again  reviewed  and  revised.  Test 
and  Evaluation  was  further  refined  in  DOD  Directives  and  Military  Standards  documents 
and  was  becoming  a  major  concern  of  the  Pentagon  and  The  Congress.  In  1983  Congress 
directed  the  establishment  of  the  Director  Of  Operational  Test  And  Evaluation  (DOT&E). 
In  1983  the  Assistant  Seaetary  Of  Defense  made  the  following  statement: 

...  the  criterion  should  not  be  how  quickly  we  can  field  any  new  weapon,  but 
rather  how  quickly  we  can  field  a  new  weapon  that  works.  The  onl^  weapons  that 
would  be  significantly  delayed  would  be  the  ones  that  operational  testing  shows  to  be 
unsuitable  for  combat,  and  1  cannot  believe  that  any  of  us  would  advocate  saddling 
our  flghting  forces  with  any  of  those.  In  fact,  the  most  likely  effect  of  operational 
testing  is  not  to  delay,  but  to  accelerate  the  development  process.  Trying  to  fix  a 
faulty  weapon  after  it's  in  the  Held  ->  if  it  can  still  be  fixed  -  is  a  far  slower  process 
than  fixing  the  design  before  it  goes  into  production.  [Ref.  5] 

Testing  continued  to  be  a  target  of  discussion  and  reform  fiom  outside  as  well  as 
inside  the  Pentagon  throughout  the  1980's  along  with  the  acquisition  process. 

4.  DOD  Acquisition  Revision 

Defense  Management  was  again  looked  at  in  1985  by  a  Presidential  Blue  Ribbon 
Commission  on  Defense  Management  and  there  were  still  other  revisions  of  the  acquisition 
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process  in  1987  and  in  1991.  Hie  focus  of  each  of  these  revisions  was  an  attempt  to  make 
the  acquisition  process  less  costly,  less  time  consuming,  and  more  responsive  to  the  needs 
of  the  operational  community.  [Ref,  6]  The  result  of  these  latest  revisions  is  our  current 
acquisition  system,  a  system  that  is  again  under  review.  Among  the  many  initiatives  of  the 
latest  revisions  was  the  push  to  consolidate  documentation.  In  the  testing  arena,  DOD 
Directive  5000.3  "Test  and  Evaluation"  was  canceled  along  with  other  5000  scries 
Directives  and  integrated  into  DOD  Directive  5000.2,  February  1991,  part  8.  Other  affects 
on  testing  include  the  requirement  of  live  fire  testing  of  major  weapon  systems  before  the 
production  phase  and  the  inclusion  of  test  information  in  various  reports,  the  Selected 
Acquisition  Report  (SAR)  for  example.  [Ref.  7]  The  acquisition  process  is  again  under 
review  and  almost  any  revisions  to  the  acquisition  process  will  likely  impact  on  testing. 

C.  TYPES  OF  T&E 

There  are  four  major  types  of  testing,  DT&E  (Development),  OT&E  (Operational), 
Multi'Service  Test  and  Evaluation  and  Joint  Test  and  Evaluation.  The  types  of  testing 
which  fall  into  the  realm  of  DT&E  or  at  least  Director,  Test  and  Evaluation  (DTE)  oversight 
include  qualification  testing.  Live  Fire  Test  and  Evaluation  (LFT&E)  and  Production 
Acceptance  Test  and  Evaluation  (PAT&E).  The  following  sections  describe  these  various 
types  of  testing.  [Ref.  8] 

1 .  Development  Test  and  Evaluation  (DT&E) 

Development  Test  and  Evaluation  (DT&E)  is  an  iterative  process  of  design,  build, 
test,  identify  deHciencies,  fu,  retest,  and  repeat.  DT&E  is  conducted  throughout  the 
acquisition  process  to  ensure  the  acquisition  and  fielding  of  an  effective  and  supportable 
system.  It  is  performed  in  the  factory,  laboratory  and  on  the  proving  ground  by  contractors 
and  the  Government.  DT&E  includes  lest  and  evaluation  of  components  and  subsystems  at 
all  Work  Breakdown  Structure  (WBS)  levels,  it  includes  modifications,  hiudware/software 
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iterations,  related  software  and  qualification  testing.  Contractor  and  Government  testing 
may  be  combined  into  one  integrated  program  test  and  conducted  to  determine  if  the 
technical  development  of  the  acquisition  process  has  been  met  as  well  as  provide  data  to  the 
decision  authority.  [Ref.  9]  DT&E  involves  the  use  of  simulations,  models,  breadboards, 
brassboards,  and  test  beds,  and  full  scale  engineering  development  models  or  prototypes  of 
system  components  or  the  system  itself.  DT&E  is  conducted  throughout  the  acquisition 
process  as  described  in  the  "Testing  and  the  Acquisition  Process"  section  of  this  chapter. 

a.  Qualification  Testing 

Qualification  testing  is  performed  to  verify  the  design  and  manufacturing 
process  and  provide  a  baseline  for  subsequent  acceptance  tests.  These  tests  include 
Production  Qualification  Tests  (PQT),  First  Article  Tests  (FAT)  and  other  down  line 
production  qualification  tests  performed  to  verify  process  control.  The  Production 
(^alificadon  Test  is  conducted  at  the  unit,  subsystem  (component)  and  system  level  on 
production  items  and  completed  before  production  decisions.  The  First  Article  Tests  (FAT) 
consist  of  a  series  of  formal  contractual  tests  conducted  to  ensure  the  effectiveness  of 
process,  equipment  and  procedures.  The  FAT  is  conducted  on  a  random  sample  from  the 
first  production  lot.  [Ref.  10] 

b.  Live  Fire  Test  and  Evaluation  (LFT&E) 

Live  Fire  Testing  was  mandated  by  Congress  in  November  1986.  The  law 
stipulates  that  major  acquisition  programs  may  not  proceed  beyond  Low  Rate  Initial 
Production  (LRIP)  until  realistic  survivability  (or  lethality  for  some  systems)  testing  has 
been  completed.  This  testing  requires  the  Services  to  Live  Fire  test  their  weapon  systems 
as  early  as  possible  before  Milestone  m.  [Ref.  1 1]  LFT&E  is  its  own  type  of  testing  and 
while  it  is  not  truly  DT&E,  the  Congress  recognized  the  importance  of  keeping  the  Live 
Fire  Program  coupled  closely  to  the  development  piocess  and  affirmed  this  by  requiring  the 
Director,  Live  Fire  Testing  to  report  directly  to  the  USDA.  [Ref.  12]  LFT&E  is  also 
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sometinies  associated  with  OT&E.  However,  there  are  significant  differences  between 
LFTAE  and  OT&E.  OT&E  is  further  described  in  the  section  "Operational  Test  and 
Evaluation"  of  this  chapter.  Figure  1  highlights  the  main  differences  between  LFT&E  and 
OT&E. 


LIVE  FIRE 

OPERATIONAL 

Full-Up  Destructive  Testing 

Typically  Nondestructive 

Instrumented  To  Gather 
Vulnerability  /Lethality  Data 

Instrumented  So  As  Not  To 
Interfere  With  Tactical 

Realism 

Typically  One  On  One 

Typically  Few  On  Few 

Oversight,  Director,  Live  Fire 
Testing 

Oversight,  Director,  OT&E 

FIGURE  1.  Live  Fire  Testing  Versus  Operational  Testing 


c.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

The  Production  Acceptance  Test  And  Evaluation  (PAT&E)  is  conducted  on 
production  items  to  demonstrate  that  those  items  meet  the  requirements  and  specifications 
of  the  procuring  contracts  so  production  may  continue.  PAT&E  also  ensures  that 
production  line  systems  demonstrate  the  same  performance  characteristics  and  capabilities 
of  the  preproduction  models.  Such  testing  is  normally  conducted  by  the  Program  Office 
quality  assurance  section,  often  at  the  contractor's  plant  and  may  involve  operational  users. 
[Ref.  13] 

2.  Operational  Test  and  Evaluation. 

The  purpose  of  Operational  Test  And  Evaluation  (OT&E)  is  to  assess  operational 
effectiveness  and  suitability  at  each  stage  in  the  acquisition  process.  Operational 
effectiveness  is  a  measure  of  the  contribution  of  the  system  to  mission  accomplishment 
under  actual  conditions  of  employment.  Operational  suitability  is  a  measure  of  the 
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mainuinability  and  reliability  of  the  system,  the  effort  and  level  of  training  required  to 
maintain,  support,  and  operate  it,  and  any  unique  logistic  or  training  requirements  of  the 
system.  OT&E  may  also  provide  information  on  tactics,  doctrine,  organization  and 
personnel  requirements  and  may  be  used  to  assist  in  the  preparation  of  operating  and 
maintenance  instructions  and  other  publications.  OT&E's  most  important  aspect  is  that  it 
provides  an  independent  evaluation  of  the  utility  of  the  system  and  the  feasibility  of 
employing  it  [Ref.  14] 

For  major  systems,  OT&E  is  normally  planned  and  conducted  by  a  major  OT&E 
field  agency  located  within  the  DOD  component.  This  Operational  Test  Agency  (OTA) 
must  be  separate  and  independent  from  both  the  developing^rocuring  agency  and  the  using 
agency.  The  OTA  is  responsible  for  managing  operational  testing,  reporting  test  results 
and  providing  its  independent  evaluation  of  the  system  being  tested  directly  to  the  Military 
Service  Chief  or  Defense  Agency  Director.  [Ref.  15]  Like  DT&E,  OT&E  also  occurs 
throughout  the  acquisition  process.  OT&E’s  role  in  this  process  is  described  in  the 
'Testing  and  the  Acquisition  Process"  section  of  this  chapter.  Figure  2  highlights  the  major 
differences  between  DT&E  and  OT&E.  [Ref.  16] 

3.  Multi'Service  Test  and  Evaluation 

Multi-Service  Test  and  Evaluation  is  T&E  conducted  on  a  system  being  acquired 
for  more  than  one  Service.  All  Services  involved  participate  with  one  designated  as  the 
lead  Service.  At  the  conclusion  of  Multi-Service  T&E,  each  participating  OT&E  agency 
submits  an  independent  evaluation  through  its  normal  channels.  The  Lead  Service  then 
prepares  a  single  report  that  reflects  the  system's  operational  effectiveness  and  suitability 
for  each  Service.  This  report  goes  forward  to  the  Defense  Acquisition  Board  (DAB)  for 
review,  recommendations  and  decision.  [Ref.  17] 
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DT&E 

OT&E 

Controlled  by  Program 
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One  on  One  Tests 

Many  on  Many  Tests 

Controlled  Environment 
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Component/sub-system 

Complex  System 

Contractor  Involvement 

No  Contractor  Involvement. 

Test  to  Specification 

Test  to  Requirements 
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Equipment 

Development  Test  Article 

Production  Representative 
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Precise  Performance 

Objectives  and  Threshold 
Measurements 

Performance  Measurement  of 
Operational  Effectiveness  and 
Suitability. 

FIGURE  2.  Differences  Between  DT&E  and  OT&E 


4.  Joint  Test  and  Evaluation 

Joint  Service  Test  and  Evaluation  is  a  specific  program  activity  sponsored  by  the 
Office  of  the  Secretary  of  Defense  (OSD).  JT&E  Programs  are  not  primarily  acquisition 
oriented.  Rather,  they  are  means  of  examining  Joint  Service  tactics,  doctrine  and  systems' 
interoperability.  JT&E  provides  information  on  system  requirements  or  improvements  and 
for  force  structure  planning.  There  are  both  Joint  Development  T&E  and  Joint  Operational 
T&E.  Joint  Developmental  T&Es  focus  on  obtaining  information  on  system  requirements, 
performance,  reliability  and  otha  technical  aspects.  Joint  Operational  T&Es  are  conducted 
to  obtain  data  pertinent  to  operational  doctrine,  tactics  and  procedures.  [Ref.  18] 
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D.  TESTING  AND  THE  ACQUISITION  PROCESS 

Testing  is  critical  throughout  the  life  cycle  of  any  system.  Both  DT&E  and  OT&E 
events  occur  throughout  the  acquisition  process  which  consists  of  the  following  phases: 

•  Concept  Exploration  and  Definitiori  •  Phase  0 

•  DemonstrationA'alidation  -  Phase  1 

•  Engineering  and  Manufacturing  Oevdopment  -  Phase  n 

•  Production  and  Deployment  -  Phase  m 

•  Operations  and  Support  Phase  rv 

Figure  3  depicts  these  phases  and  illustrates  that  the  phases  are  separated  by  key 
decision  points  or  milestones.  These  decision  points  occur  throughout  the  program  life 
when  a  decision  authority  reviews  a  program  and  authorizes  it  to  advance  to  the  next  phase 
in  the  cycle.  The  following  section  describes  the  acquisition  process  and  testing  events 
normally  occurring  during  the  respective  phase. 

1 .  Concept  Exploration  and  Definition  Phase  *  PHASE  0 

The  defense  system  acquisition  process  begins  with  the  submission  of  a  Mission 
Need  Statement  (MNS)  with  the  Service's  Program  Objective  Memorandum  (POM).  The 
MNS  documents  major  mission  deficiencies  (or  improvement  opportunities)  in  a  Service's 
ability  to  meet  its  mission  requirements.  The  Concept  Exploration  and  Definition  Phase 
(C/E)  follows  the  Milestone  0  approval  for  concept  studies.  During  the  (C7E)  Phase 
alternative  approaches  for  satisfying  the  requirement(s)  are  investigated.  Concept 
Exploration/Definition  phase  assists  in  selecting  preferred  alternative  system  concepts, 
technologies,  and  designs.  Documents  for  the  Milestone  I  review  are  the  MNS,  the 
Acquisition  Decision  Memorandum  (ADM)  which  provides  the  exit  criteria,  the  Operational 
Requirements  Document  (ORD)  which  delineates  the  qualitative  and  quantitative  system 
parameters  and  the  Test  and  Evaluation  Master  Plan  (TEMP)  which  identifies  the 
objectives,  responsibilities,  resources  and  schedule  for  the  T&E  effort 
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IS 


BEYOND  UUP 


a.  DT  0 


Early  in  this  phase  a  review  of  historical  tests  and  existing  systems  is 
conducted.  Preferred  alternative  system  concepts  are  selected  and  a  draft  Test  and 
Evaluation  Master  Plan  (TEMP)  is  started.  The  TEMP  defines  test  phases,  schedules  and 
resource  requirements.  Prior  to  the  Milestone  I  decision,  laboratory  testing,  modeling  and 
simulations  are  conducted  by  the  contractor  and/or  the  Government  development  agency  to 
demonstrate  and  assess  the  capabilities  of  key  subsystems  and  components.  [Ref.  19] 
b.  OTO 

The  operational  test  agency  (OTA)  estimates  military  utility  and  assesses 
operational  impact  of  candidate  technical  approaches.  The  operational  test  agency  also 
monitors  CTE  Test  and  Evaluation  for  future  test  planning.  OT&E  conducted  during  this 
phase  supports  developing  estimates  of  the  need  for  the  new  system,  a  sound  physical 
basis  for  the  new  system,  the  system's  affordability  and  the  impact  of  the  system  on  the 
force  structure.  [Ref.  20] 

2.  Demonstration/Validation  Phase  •  PHASE  I 

After  the  Milestone  1  decision,  the  program  enters  the  Concept 
Demonstration/Validation  (DEM/VAL)  Phase  during  which  selected  concepts,  typically 
brassboard  or  early  prototype,  are  reHned  through  study  and  analysis.  [Ref.  21]  This 
phase  ends  with  the  Milestone  II  decision  to  either  enter  into  Engineering  and 
Manufacturing  Development  (EMD),  conduct  more  research  and  development  and  delay  the 
decision  or  tenninate  die  program.  Documents  of  particular  interest  to  the  T&E  manager  at 
die  time  of  the  Milestone  U  review  include  the  ADM,  an  updated  TEMP,  the  updated  ORD, 
die  Cost  and  Operational  Effectiveness  Analysis  (COEA),  which  is  a  cost  and  operational 
analysis  of  the  alternative  systems,  and  die  Development  Baseline. 
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a.  DTI 


During  this  phase  DT&E  is  accomplished  to  ensure  that  engineering  is 
reasonably  complete,  demonstrate  technical  risk  areas  have  been  identified  and  can  be 
reduced,  identify  the  best  or  prefened  technical  approach  and  that  the  concept  can  meet 
operational  requirements.  DT 1  includes  T&E  of  components,  subsystems,  and  prototype 
development  models.  Testing  also  includes  functional  compatibility  and  interoperabilty 
with  existing  and  planned  equipment.  DT  conducted  during  this  phase  is  most  often 
conducted  at  the  contractor's  facility  with  Government  oversight  [Ref.  22] 
b.  OT  I 

In  OT 1  the  OT&E  agency  prepares  independent  early  operational  assessments 
to  identify  the  best  design,  indicate  the  risk  level  of  performance  foi  this  phase  of 
development,  and  estimate  potential  operational  effectiveness  and  suitability.  The 
operational  aq)ects  of  the  technical  approaches  is  examined  and  information  on  tactics, 
doctrine,  organization,  personnel  requirements  and  critical  issues  are  identified.  The  OTA 
also  identifies  needed  modifications  or  other  issues  that  need  to  be  resolved  before  the  next 
phase  is  initiated.  Typical  operational  and  support  personnel  are  used  to  obtain  an  estimate 
of  the  user's  capability  to  operate  the  system.  The  OT&E  assessments  provide  a  record  of 
testing,  an  audit  trail,  test  data,  recommendations  and  conclusions.  Testing  normally 
includes  components,  subsystems,  brassboard  configurations  or  advanced  development 
prototypes.  [Ref.  23] 

3 .  Engineering  and  Manufacturing  Development  Phase  •  PHASE  n 

The  objective  of  the  Engineering  and  Manufacturing  Development  (EMD)  Hiase  is 
to  design,  fabricate  and  test  a  preproduction  system  that  closely  approximates  the  final 
product.  This  phase  may  include  Live  Fire  Testing  if  required.  The  information  from  the 
DT  and  OT  along  with  other  documents  such  as  the  Updated  TEMP,  the  Beyond-Low  Rate 
Initial  Production  Report  (LRIP)  and  a  Live  Fire  Test  Report  (The  Beyond  LRIP  Report 


17 


and  Live  Fire  Test  Report  are  required  by  law  of  the  Director,  Operational  T&E)  provide 
the  infonnation  to  the  decision  makers  for  determining  whether  to  enter  production  or  not 
and  what  level  of  production.  [Ref.  24]  Data  obtained  during  EMD  test  and  evaluation  are 
used  to  assist  in  evaluating  the  system's  maintenance  training  requirements  and  in 
evaluating  the  proposed  training  program.  Test  results  generated  during  EMD  also  support 
the  user  in  refining  and  updating  tactics.  [Ref.  25] 

a.  DT  a 

DT  II  must  demonstrate  that  engineering  is  reasonably  complete,  that  all 
significant  design  solutions  to  problems  are  in  hand,  and  that  the  design  meets  its  required 
specifications  within  the  range  of  environmental  limits  designed  for  the  operational 
employment  of  the  system.  DT  II  also  must  verify  "fixes"  from  DT  I  and  assess  the 
survivability,  vulnerability  and  logistic  supportability  of  the  system.  Vulnerability  (or 
lethality)  may  require  live  fire  testing. 

The  final  phase  of  DT  II  is  the  TECTIEVAL.  The  TECHEVAL  is  the  formal 
demonstration  that  the  design  meets  specifications  and  it  provides  the  major  source  of  data 
for  certification  of  readiness  for  the  OPEVAL.  The  TECHEVAL  provides  information 
relative  to  the  technical  performance  of  the  system.  It  is  the  qualification  of  components 
and  an  assessment  of  compatibility,  inter-operability,  vulnerability,  lethality, 
transportability,  etc.  The  technical  evaluation  also  determines  performance  limitations  and 
safe  operating  parameters,  insures  the  effectiveness  of  the  manufacturing  process  and 
confirms  readiness  for  operational  testing.  [Ref.  26]  Typical  test  models  for  this  phase 
include  pre  production  prototypes  or  pilot  production  models. 

b.  OTJJ 

OT II  is  conducted  to  demonstrate  performance  of  the  program  objectives.  It 
estimates  operational  effectiveness  and  suitability  as  well  as  identifies  operational 
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deficiencies.  OT  D  is  used  to  determine  adequacy  of  publications  and  support  equipment 
and  to  provide  information  to  refine  operations  and  support  (O&S)  cost  estimates. 

The  final  phase  of  OT  II  is  the  OPEVAL.  The  Operational  Evaluation 
(OPEVAL)  occurs  a  minimum  of  90  days  after  the  TECHEVAL.  It  assists  the  developers 
by  providing  information  relative  to  operational  performance,  doctrine,  tactics,  training  aritl 
logistical  issues.  It  assists  the  decision  makers  on  the  overall  suitability  of  the  system  to  be 
delivered  as  well  as  bfluences  either  a  low  rate  initial  production  or  a  full-rate  production. 
The  OPEVAL  also  assesses  the  user's  viewpoint  on  the  system’s  desirability.  [Ref.  27] 
Typical  test  models  for  this  phase  include  preproduction  prototypes  or  pilot  production 
models. 

4.  Production  &  Deployment  •  PHASE  III 

The  objective  of  this  phase  is  to  produce  and  field  the  system.  Production 
Acceptance  Test  and  Evaluation  (PAT&E)  is  conducted  on  production  items  to  ensure  the 
effectiveness  of  the  manufacturing  process,  equipment,  and  procedures.  Follow-on 
Operational  Testing  (FOT&E)  may  be  conducted  to  verify  operational  effect  veness  and 
suitability.  [Ref.  28] 
a.  DT  in 

After  the  Milestone  III  (Production  and  Deployment)  decision,  Developmental 
Testing  remains  an  integral  part  of  the  development,  validation,  and  introduction  of  system 
changes  undertaken  to  improve  the  system  or  to  reduce  life  cycle  costs.  [Ref.  29]  DT  III 
verifies  conections  of  deficiencies  in  the  TECHEVAL,  verifies  specification  compliance 
and  completes  any  testing  not  completed  during  EMD.  DT  m  is  also  used  to  conduct  the 
major  elements  of  the  PAT&E.  The  PAT&E  ensures  that  production  line  systems 
demonstrate  the  same  performance  charactnistics  of  the  preproduction  models.  PAT&E 
will  continue  throughout  the  production  life  cycle  of  the  system  Testing  conducted  in  this 
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phase  is  conducted  under  controlled  conditions,  provides  quantitative  and  qualitative  data 
and  is  normally  monitcred  or  conducted  by  a  Government  representative.  [Ref.  30] 
b,  OT  in 

arm  is  used  to  verify  correction  of  OPEVAL  deficiencies  and  to  evaluate 
performance  not  tested  during  earlier  tests.  OT  III  is  conducted  on  the  OPEVAL  model 
with  fixes.  OT  in  takes  the  form  of  Follow-on  Test  and  Evaluation  (FOT&E)  and  is 
conducted  with  production  articles  in  operational  organizations.  This  testing  verifies  the 
production  system,  tests  operational  effectiveness  and  suitability  under  realistic  operational 
conditions  and  demonstrates  reliability  and  maintainability  improvements.  [Ref.  31] 

5 .  Operations  and  Support  •  PHASE  IV 

The  function  of  this  phase  is  to  ensure  that  the  fielded  system  continues  to  provide 
the  capabilities  required  and  to  identify  the  actions  and  resources  needed  to  maintain 
operational  readiness  and  support  objectives.  This  phase  ends  with  a  Major  Modification 
Approval  to  identify  the  actions  and  resources  needed  to  achieve  and  maintain  operational 
readiness  and  support  objectives.  The  Major  Modification  Approval  encompasses  a  review 
of  a  system's  operational  effectiveness,  suitability,  and  readineiis  to  determine  whether 
major  upgrades  are  necessary  or  deflciencies  warrant  consideration  of  replacement.  In 
preparation  for  this  milestone  the  TEMP,  and  prociuct  baseline  are  updated  to  describe 
program  status,  changes  and  issues.  [Ref.  32] 
a.  DT 

Development  testing  during  this  phase  ensures  previous  test  deficiencies  are 
corrected.  DT  evaluates  proposed  production  improvements.  Engineering  Change 
Proposals  (ECPs),  upgrades  and  determines  if  the  resources  are  available  to  maintain 
readiness  and  support  objectives  throughout  the  system's  acquisition  life  cycle.  As  the 
system  completes  its  useful  life,  DT  is  used  in  an  engineering  aspect  to  help  modify  the 
system  for  new  threats  or  with  new  technology  or  to  help  in  system  disposal. 
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h.  OT  IV 

A  major  function  of  OT  during  this  phase  is  to  evaluate  post  production 
logistic  readiness  and  support  and  to  validate  effectiveness  and  suitability  of  modified 
systems.  Follow-on  Test  and  Evaluation  (FOT&E)  are  used  to  assess  logistics  readiness, 
sustainability,  and  the  implementation  of  the  Integrated  Logistics  Support  Plan  (ILSP). 
Finally  as  a  system  approaches  the  end  of  its  usefulness,  OT  monitors  the  system's  current 
state  of  operational  effectiveness,  suitability  and  readiness  to  determine  whether  major 
modifications  are  necessary  or  deficiencies  warrant  consideration  of  replacement  [Ref.  33] 

E.  DT&E  IN  THE  UNITED  STATES  ARMY 

This  section  describes  the  testing  structure  starting  from  the  Office  of  the  Secretary  of 
Defense  (OSD)  down  to  the  key  players  involved  in  Army  DT&E. 

1 .  DOD  Test  Structure 

The  organizational  structure  of  the  DOD  concerning  acquisition  and  testing  is 
depicted  in  Figure  4.  T&E  oversight  is  performed  by  two  offices:  the  Director,  Test  and 
Evaluation  (DTE)  and  the  Director,  Operational  Test  and  Evaluation  (DOT&E)  The 
Defense  Acquisition  Executive  (DAE),  a  position  held  by  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology  (USD  (A&T)),  performs  the  management  of  acquisition 
for  the  DOD. 

The  DTE  is  the  principal  staff  assistant  and  advisor  to  the  USD  (A&T)  for  T&E 
matters.  The  DTE  is  responsible  for  all  DT&E.  The  duties  of  the  DTE  include:  review 
major  acquisition  program  documentation  for  DT&E  in^lications,  provide  management  and 
oversight  of  the  major  ranges  and  test  facilities,  and  develop  and  implement  the  Live  Fire 
Test  Program. 

The  DOT&E  reports  directly  to  the  Secretary  of  Defense  (SECDEF)  and  has 
special  reporting  requirements  to  the  Congress.  The  DOT&E's  responsibility  is  to  provide 
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unbiased  insight  into  the  operational  effectiveness  and  suitability  of  new  weapon  systems. 
The  duties  of  the  DOT&E  include  approving  test  plans  on  major  systems  prior  to  OT&E, 
approval  of  OT&E  funding  for  major  systems  and  providing  the  SECDEF  and  the 
Congress  with  the  Beyond  LRIP  report.  [Ref.  34] 

2 .  Army  T&E  Structure 

The  Army  management  structure  for  T&E  is  illustrated  in  Figure  5.  The  Under 
Secretary  of  the  Army  is  the  Army  Acquisition  Executive  (AAE).  The  AAE  is  responsible 
for  all  acquisition  T&E  (operational  and  developmental  tests)  planning,  programminp, 
budgeting,  developmental  test  policy  and  oversight.  The  AAE  performs  these  duties  with 
the  assistance  of  the  Assistant  Secretary  of  the  Army,  Research,  Development,  and 
Acquisition  (AS  A/RD  A).  The  AS  A/RD  A  is  organized  to  provide  technical  assessments  and 
program  evaluations.  He  resolves  acquisition  issues  whenever  possible  and  makes 
recommendations  to  the  AAE  on  the  acquisition  of  weapon  systems.  The  Deputy  Under 
Secretary  of  the  Army  for  Operations  Research  (DUSA(OR))  is  chartered  to  supervise  all 
Army  T&E  policy  and  has  oversight  for  all  Army  T&E.  [Ref.  35] 

3 .  Army  DT&E 

The  U.  S.  Army  Materiel  Command  (AMC)  is  responsible  for  the  management  of 
DT&E.  Under  AMC  the  Test  and  Evaluation  Command  (TECOM)  has  the  primary 
responsibility  for  conducting  developmental  tests  for  the  Army  and  the  Army  Materiel 
Systems  Analysis  Agency  (AMSAA)  conducts  test  analysis  and  evaluation  on  major 
systems.  TECOM  may  be  designated  as  the  evaluator  for  non  major  systems.  The 
Structure  of  AMC  and  TECOM  is  shown  in  Figure  6. 

TECOM  is  responsible  for; 

•  Planning,  executing  and  rei)orting  the  results  of  technical  tests.  Technical  tests  include 
Development  Tests,  Technical  Feasibility  Tests,  Production  (Salification  Tests,  Joint 
Tests,  and  contractor/foreign  tests. 

•  Providing  test  facilities  and  technical  expertise  in  support  of  the  T&E  life  cycle. 
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FIGURE  5.  Army  T&E  Structure 


FIGURE  6.  AMC  and  TECOM  Structure 
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CTECOM  responsibilities  continued) 

•  Maintaining  the  Army's  T&E  data  base. 

•  Maintaining  the  Army’s  Major  Range  and  Test  Facility  Base. 

•  Researching,  developing,  and  acquiring  instrumentation  and  developing  new  and 
improved  test  methodology. 

•  Providing  safety  confirmations.  [Ref.  36] 

F.  OTHER  DT&E  ORGANIZATIONS  IN  THE  ARMY. 

The  testing  process  extends  throughout  the  acquisition  cycle  of  any  system.  Many 
agencies  play  a  significant  role  in  the  testing  process.  The  Program  Management  Office 
(PMO),  the  Tester  and  his  facilities,  the  builder  or  Contractor,  the  evaluator,  the  Army 
Materiel  Systems  Analysis  Agency  (AMSAA)  in  the  Army's  case  and  the  user  or  Combat 
Developer  all  are  a  part  of  DT&E.  They  meet  formally  through  the  Test  Integration 
Working  Groups  (TIWG)  and  informally  based  on  other  factors  such  as  critical  issues  or 
Program  Management  style. 

1 .  Program  Management  Office  (PMO) 

The  Program  Manager  (PM)  is  ultimately  responsible  for  all  aspects  of  the  system 
development,  to  include  coordinating  the  total  T&E  program.  The  PM  normally  has  a 
deputy  or  assistant  whose  functions  include  the  overwatch  of  testing  as  well  as  writing  or 
aputting  into  various  test  documents  and  reports.  The  PM  and  his  office  arc  responsible 
•'or  writing  numerous  reports  and  plans  such  as  the  Test  and  Evaluation  Master  Plan 
(TEMP).  The  input  to  the  TEMP  is  normally  influenced  by  the  Test  Integration  Working 
Group. 

a.  Test  Integration  Working  Croup  (TIWG) 

The  PM  charters  and  uses  the  Test  Integration  Working  Group  (TIWG)  to 
help  coordinate,  plan,  and  discuss  die  testing  and  analysis  effort.  TIWG  members  include 
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representatives  froni  the  development  agency,  the  user,  both  developmental  and  operational 
T&E  agencies,  logistics,  analysis  and  training  organizations.  [Ref.  37] 

2.  TECOM 

The  U.  S.  Army  Test  and  Evaluation  Command  (TECOM)  is  the  Army's  testing 
agency.  TECOM  as  presented  in  Figure  6  has  facilities  throughout  the  U.  S.  as  well  as 
locations  outside  of  the  continental  U.  S.  As  sub  components  of  TECOM,  these  facilities 
provide  the  people,  equipment,  and  other  resources  to  conduct  various  types  of  DT. 
TECOM  through  its  various  facilities  is  responsible  for  the  planning,  executing  and 
reporting  the  results  of  technical  tests.  Technical  tests  include  Development  Tests, 
Technical  Feasibility  Tests,  Production  Qualification  Tests,  Joint  Tests,  and 
contractor/foreign  tests.  TECOM  is  charged  with  maintaining  the  Army’s  Major  Range  and 
Test  Facility  Base,  maintaining  the  Army's  T&E  data  base  and  researching,  developing, 
and  acquiring  instnunentation  and  improved  test  methodology. 

3.  Prime  Contractor 

The  Prime  Contractor  is  the  Company  who  is  responsible  to  provide  the 
Government  with  the  needed  product.  The  Prime  Contractor  plays  a  significant  role  in 
DT&E.  The  contractor  conducts  his  own  testing  prior  to  Government  tests  and 
demonstrates  that  he  is  prepared  to  enter  into  Government  conducted  or  at  least 
Government  observed  testing.  The  contractor’s  testing  during  the  initial  phases  of  the 
acquisition  cycle  is  likely  to  impact  testing  conducted  by  the  Government  during  the  EMD 
Phase.  The  contractor  may  even  conduct  some  DT,  observed  by  the  Government,  within 
his  facility. 

4.  AMSAA 

The  Army  Materiel  Systems  Analyas  Agency  is  AMC’s  independent  evaluator  for 
the  DT  process  often  referred  to  as  the  "honest  broker."  AMSAA  is  responsible  for  the 
Independent  Evaluation  Plan  (lEP)  as  well  as  advising  the  tester  and  the  program  office  on 


analytical  issues,  testing  and  test  documentation.  AMSAA's  greatest  influence  is  on  the 
statistical  process  controls  of  the  tests  such  as  sample  size  and  confidence  levels  and  test 
design.  AMSAA  conducts  the  analysis  and/or  evaluation  of  the  testing  and  provides 
members  to  Ac  TTWG. 

5.  Combat  Developer 

The  Combat  Developer  is  usually  the  “user”  or  the  organization  that  represents  the 
user  and  identifles  the  need  for  the  system  being  developed.  In  the  Army  this  is  usually  a 
Training  and  Doctrine  Command  (TRADOC)  basic  Army  branch  such  as  the  Field  Artillery 
or  the  Armor  School.  These  agencies  develop  the  doctrine  and  training  for  their  respective 
branches  based  on  overall  Army  tactics,  doctrine  and  guidance.  They  also  help  to 
determine  the  needs  of  that  particular  branch  presently  and  into  the  future.  These 
organizations  provide  the  Mission  Need  Statements  that  may  grow  into  development  of  a 
new  system  or  modification  of  an  old  system.  In  some  cases  the  user  may  be  The 
Combined  Arms  Center  (CAC)  or  The  Combined  Arms  Service  Support  Center  (CASSC). 
These  agencies  integrate  the  training,  doctrine  and  needs  of  the  various  branches  they 
represent  and  function  as  an  overall  user  for  a  system  needed  by  all  the  combined  branches. 
The  Combat  Development  agency  is  the  organization  where  the  acquisition  process  begins 
and  where  the  final  product  arrives. 

G .  POTENTIAL  ISSUES  AND  PROBLEMS 

There  are  potentially  many  issues  or  problems  that  can  affect  E>evelopmental  Testing 
and  many  issues  that  are  affected  by  DT&E.  This  chapter  provided  a  basic  history  of 
testing  in  the  Department  of  Defense,  described  testing  within  the  acquisition  process  and 
the  Developmental  Testing  function  within  the  United  States  Army.  The  structure  of  the 
testing  within  the  DOD  and  within  the  Army  is  effected  by  legislation,  policy,  both  formal 
and  informal  guidance  and  a  continuous  reform  effort.  The  management  and  conduct  of 
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DT&E  impacts  upcoming  milestone  decisions  and  may  hold  the  key  to  a  program's  success 
and  continuation  or  its  cancellation.  The  problems  that  occur  in  DT&E  impact  on  cost, 
schedule  and  performance  of  the  system  and  are  very  important  to  both  the  Government 
agencies  and  the  contractors  involved.  The  next  chapter  identiHes  those  issues  that  are 
considered  the  most  prevalent  according  to  the  major  players  in  a  number  of  programs. 


III.  METHODOLOGY  AND  DATA  SUMMARY 


A.  INTRODUCTION 

This  chapter  presents  the  methodology  used  and  data  gathered  to  answer  the  primary 
and  subsidiary  thesis  questions.  An  overview  of  the  systems  investigated  is  presented 
along  with  a  summary  of  those  data.  Research  investigation  included  a  literature  search  of 
After  Action  Reports  from  Test  and  Evaluation  Command  (TECOM),  lessons  learned 
reports  of  major  systems,  General  Accounting  Office  Reports,  Congressional 
Subcommittee  Reports,  Developmental  Test  and  Evaluation  Reports  and  technical  and 
professional  journals  and  manuals.  Interviews  were  conducted  with  program  office 
personnel,  program  testers,  analysis  personnel,  user  representatives  and  contractors  who 
have  participated  in  the  developmental  testing  of  the  major  programs  selected.  Interviews 
were  also  conducted  with  personnel  who  had  years  of  experience  in  the  area  of 
developmental  testing.  Interviews  were  conducted  in  person  and  over  the  phone. 

B.  METHODOLOGY 

The  focus  of  the  literature  search  and  the  interviews  was  to  address  the  primary  thesis 
question.  The  primary  thesis  question  was: 

What  is  the  most  significant  problem  in  conducting  developmental  testing? 

1 .  Basic  Interviews 

Interviews  were  the  primary  method  of  addressing  the  research  question.  A  scope 
and  limitation  for  the  interviews  were  developed  in  order  to  organize  the  information. 
About  a  dozen  systems  were  considered,  this  was  reduced  to  seven.  The  main 
Government  agencies  dealing  with  DT&E  were  interviewed.  These  interviews  included 
representatives  from  the  Program  Office,  the  tester,  the  analyst  from  AMSAA,  a  user 
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representative  or  the  Combat  Developer  and  the  Prime  Contractor.  A  basic  set  of  questions 
was  developed  and  reviewed  by  students  and  instructors  with  test  and  evaluation 
backgrounds.  The  initial  foimat  was  general  for  all  potential  interviewees  and  looked  at 
these  areas: 

•  What  do  you  consider  the  primary  problem  or  issue  in  Developmental  Testing? 

•  What  can  your  agency  or  any  of  the  others  you  work  >vith  do  about  the  problem?  To 
ascertain  the  "work  with"  relationship  of  the  various  agencies  and  offices,  the 
questionnaire  also  focused  on: 

•  How  do  you  (your  office)  interface  with  the  other  agencies?  Is  this  sufficient? 

The  questions  further  evolved  into  five  separate  formats,  similar  overall,  but 
tailored  to  the  particular  agency  being  addressed.  For  example  a  PM  office  was  asked  how 
a  particular  problem  affected  schedule  or  cost.  AMSAA  would  be  asked  instead  how  the 
problem  affected  their  analysis  or  reporting.  In  most  cases  the  interviewee  was  given  a 
draft  of  the  questionnaire  or  allowed  to  thoroughly  answer  all  questions  before  any  direct 
questioning.  A  copy  of  the  questionnaire  for  the  Program  Management  Office  is  provided 
in  Appendix  A. 

2.  Special  Interviews 

Other  interviews  were  conducted  to  gain  insight  from  people  with  background  and 
experience  in  DT&E.  These  people  were  supervisors,  branch  and  division  chiefs  at  the  test 
facilities,  at  AMSAA,  at  TECX)M  and  within  the  TRADOC  System  Management  (TSM) 
Offices.  Using  the  same  questionnaire  format  and  focusing  on  the  primary  thesis  question 
these  individuals  were  also  interviewed.  Again,  they  normally  had  time  to  prescreen  the 
questions  or  responded  in  writing  when  time  was  limited. 

C.  SYSTEMS  RESEARCHED 

The  systems  researched  were  Acquisition  Category  (ACAT)  I  and  D  systems  tested  at 
United  States  Army  Test  and  Evaluation  Coirunand  (TECOM)  facilities  within  the  United 
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States.  Seven  programs  under  test  or  tested  in  the  past  10  years  were  researched  and 
analyzed  through  interviews  and  reports.  The  systems  include  the  Abrams  Main  Battle 
Tank  Block  n  upgrade  (Ml  A2);  Anti  Armor  Weapon  System  Medium  (AAWS-M),''the 
Javelin;"  Enhanced  Position  Location  Reporting  System  (EPLRS);  the  Avenger,  a 
mounted  Air  Defense  system,  the  Kiowa  Warrior,  an  aimed  scout  helicopter;  the  Maneuver 
Control  System  (MCS),  a  command  and  control  (C2)  system;  and  the  Family  of  Medium 
Tactical  Vehicles  (FMTV).  These  programs  represent  different  types  of  systems  from 
electronic/data  communications  and  software  to  major  weapon  systems.  They  also 
represent  different  types  of  developments,  from  system  upgrades  and  Non-Developmental 
Items  (NDI)  to  full  scale  developments. 

1 .  Abrams  Tank  Block  II  Improvement  (M1A2) 

The  Ml A2  is  the  M 1 A1  (main  battle  tank)  with  improvements  referred  to  as  the 
Block  11  Improvement  Program.  The  improvements  to  the  main  battle  tank  consist  of  an 
Improved  Commander's  Weapon  Station  (ICWS);  Commander's  Independent  Thermal 
Viewer  (CITV);  Position  Navigation  System  (POS/NAV);  and  the  core  tank.  The  core 
tank  includes  the  turret,  hull,  fire  control  electronic  units,  a  data  bus  to  interconnect  the  new 
mission  hardware;  dual  stabilization  of  the  gunner's  primary  sight  head  mirror,  the  Single 
Channel  Ground  Airborne  Radio  System  (SINCGARS);  Inter  Vehicular  Information 
System  (IVIS);  and  onboard  built-in  test  equipment.  The  M1A2  is  designed  to  provide 
Armor  and  Mechanized  units  with  improved  mobility,  protection  and  both  internal  and 
external  C3I.  [Ref.  38] 

2.  Kiowa  Warrior  (OH.S8  D) 

The  OH-58D  is  a  modernization  of  the  OH-S8A  airframe.  The  modernization 
included  a  four  blade  main  rotor  system,  an  advanced  cockpit  display  system  and  a  mast 
mounted  sight  to  provide  day/night  targeting  capability.  Armament  was  added  to  some  of 
these  helicopters  for  a  special  mission  and  eventually  resulted  in  a  modification  program  to 
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for  all  OH-58DS.  The  Kiowa  Warrior  was  designed  to  provide  reconnaissance,  security 
and  target  acquisition  functions.  It  is  used  with  Divisional  aviation,  in  support  of  armor 
assets  as  well  as  against  threat  armor  as  part  of  a  "Tank  Killer  Team"  and  employed  by 
special  operations  units.  [Ref.  39] 

3 .  Family  of  Medium  Tactical  Vehicles  (FMTV) 

The  new  Family  of  Medium  Tactical  Vehicles  is  being  designed  to  replace  the 
Army's  aging  fleet  of  Two  and  a  half  (2.5)  ton  and  Five  (5)  ton  vehicles.  These  types  of 
vehicles  are  used  for  tactical  mobility,  supply  and  support  operations.  The  vehicles  will 
include  a  2.5  ton  cargo  model,  a  5  ton  cargo  model  and  special  purpose  vehicles  such  as 
tankers,  dump  trucks  and  wreckers.  Its  expected  improvement  over  the  current  fleet  of 
vehicles  includes  greater  horsepower  and  speed,  increased  towing  capability,  higher 
reliability  and  a  high  commonalty  of  parts  among  the  various  versions.  [Ref.  40] 

4.  Enhanced  Position  Location  Reporting  System  (EPLRS) 

EPLRS  provides  the  Army  with  data  communications  and  ranging  information. 
EPLRS  reports  position  location  and  identifleation  data  on  ground  and  airborne  units  to  the 
radio  station  operator  in  near  real  time.  Its  performance  advantages  include  rapid  response 
times  and  effective  data  throughput.  Communications  Security  (COMSEC),  resistance  to 
Electronic  Couniermeasures  (ECM),  and  Electronic  Support  Measures  (ESM),  low  levels 
of  mutual  interference,  transmissions  for  ranging  measurement  and  freedom  from  voice 
data  contention.  EPLRS  will  normally  be  deployed  in  the  Division  and  Corps  areas  and 
operated  by  Army  Signal  Corps  personnel.  [Ref.  41] 

5 .  Anti  Armor  Weapon  System  -Medium  (AAWS-M),*'The  Javelin" 

The  Javelin  is  a  man  portable  antitank  weapon  system.  It  consists  of  a  round,  a 

Command  Launch  Unit  (CLU),  training  devices  and  test  equipment.  The  Javelin  has  the 
capability  to  defeat  the  current  and  projected  armor  threat  in  all  battlefield  environments  to 
include  electronic  and  electro  optical  countermeasures  and  the  electromagnetic 


environments.  The  system  provides  the  gunner  with  increased  survivability  by  having  a 
greater  range,  a  reduced  signature  and  increased  lethality.  [Ref.  42] 

6.  Pedestal  Mounted  Stinger  (PMS),  "The  Avenger" 

The  Avenger  is  an  Air  Defense  Weapon  that  has  the  requirements  of  protecting 
fiiendly  critical  assets  and  inflicting  maximum  attrition  on  threat  aircraft.  It  consists  of  a 
fire  control  unit  module  that  includes  a  turret  with  vehicle  mounted  launchers  and  a  heavy 
machine  gun.  The  system  is  mounted  on  a  High  Mobility  Multipurpose  Wheeled  Vehicle 
(HMMWV)  but  can  be  operated  remotely.  The  Avenger  has  a  Forward  Looking  Infrared 
(FLIR)  sensor,  a  laser  range  finder  and  an  onboard  computer  which  provides  the  gunner 
with  displays  to  engage  the  target,  monitor  the  system,  and  receive  and  display  command 
and  control  C2I  data.  The  Avenger  is  designed  to  protect  the  rear  areas  of  the  Corps,  the 
Divisions  and  Regiments.  [Ref.  43] 

7.  Maneuver  Control  System  (MCS) 

MCS  is  a  combination  of  hardware  and  softw.wre  intended  to  provide  commanders 
of  all  maneuver  elements  (corps  through  battalion/squadron  and  selected  companies)  a 
single  command  and  control  (C2)  system.  It  is  one  of  five  C2  systems  that  make  up  the 
Army  Tactical  Command  and  Control  System  (ATCCS).  It  includes  a  Lightweight 
Computer  Unit  (LCU),  Large  Scale  Printer  Plotter  (LSPP),  Large  Screen  Display,  Tactical 
Scanner  (TACSCAN)  and  software.  MCS  will  enhance  decision  making  and 
synchronization  among  maneuver  elements  and  as  a  part  of  ATCCS  help  integrate  and 
coordinate  other  battlefield  functions  such  as  Artillery,  Air  Defense  and  support  functions. 
[Ref.  44] 

8.  System  Categories 

The  systems  listed  above  could  be  broken  down  by  various  criteria.  To  enhance 
analysis  and  address  the  thesis  question  they  were  broken  down  by  type  of  system,  namely 
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ground  vehicle,  aircraft  etc.  The  systems  were  also  broken  down  by  acquisition  or 
development  strategy  such  as  upgrade  or  NDI. 

a.  Type  of  System 

The  systems  above  include  one  rotary  wing  aircraft,  the  OH-58D;  one  man 
portable  missile  system,  the  Javelin;  three  ground  vehicle  systems  including  a  tracked 
vehicle,  the  Ml  A2;  a  small  wheeled  vehicle  with  a  Air  Defense  missiles,  the  Avenger;  and 
family  of  wheeled  vehicles  in  different  configurations,  FMTV.  The  systems  above  also 
include  two  communication/data  and  information  systems,  MCS  and  EPLRS. 

b.  Type  of  Development 

From  a  development  standpoint  the  M1A2  Abrams  is  an  upgrade  to  the  MlAl 
tank.  The  Kiowa  Warrior  is  also  an  upgrade  to  the  OH-58A  scout  helicopter.  Both  the 
Avenger  and  the  FMTV  are  considered  Non-Developmcntal  but  for  slightly  different 
reasons.  The  Avenger  is  using  mostly  developed  technology  designed  for  the  military  use 
while  the  Family  of  Medium  Tactical  Vehicles  is  pushing  the  use  of  commercial 
components  and  parts.  The  Javelin  is  full  scale  development  weapon  system  as  are  the 
software  and  electronic  intensive  MCS  and  EPLRS. 

D.  DATA  SUMMARY 

The  data  summary  includes  information  from  both  interviews  and  literature  searches. 
The  interview  data  will  be  broken  down  by  program  and  further  broken  by  agency  or 
office.  In  order  to  obtain  answers  beyond  the  "party  line"  some  interviews  were  conducted 
under  the  premise  that  the  interviewee  by  name  would  not  be  associated  with  a  particular 
program  or  agency.  Tjierefore  the  data  are  i»’esented  by  system  but  will  not  identify  which 
system  specifically.  The  person  or  office  making  the  response  about  that  system  will  only 
be  identified  as  a  representative  of  that  agency  or  office  who  played  a  role  in  the  system's 
DT&E.  While  few  people  actually  requested  anonymity,  a  single  name  or  the  name  of  the 
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system  would  easily  divulge  almost  every  person's  identity  to  someone  who  is  familiar 
with  the  agency  or  system  described.  Each  of  the  following  sections  will  summarize  the 
respondents'  answers  to  the  questionnaire,  specifically: 

*  What  do  you  consider  the  primary  problem  or  issue  in  Developmental  Testing? 

*  What  can  your  agency  or  any  of  the  others  you  work  with  do  about  the  problem? 

*  How  do  you  (your  office)  interface  (work  with)  with  the  other  agencies? 

1 .  System  t 

a.  Program  Management  Office 

This  program  management  representative  stated  that  one  of  the  major 
problems  entering  a  Developmental  Test  was  meeting  the  test  stan  milestone  with  all 
compliant  system  hardware,  software  and  support  to  conduct  the  testing.  Delays  in  getting 
to  the  test  start  point  (caused  by  various  factors)  cause  a  test  delay  and  "all  delays  impact 
cost." 

The  program  manager  representative  determined  that  to  deal  with  this 
problem,  the  PMO,  particularly  the  PM,  require  intensive  proactive  management  at  all 
levels.  Issues  need  to  be  identified  before  they  happen  and  alternative  plans  developed.  He 
indicated  that  the  tester  "...needs  to  make  sure  resource  requirements  are  on  hand,  plus  be 
in  a  positive  position  to  adjust  to  changes."  Finally  he  noted  that  all  the  entities  involved  in 
DT&E  need  to  be  more  proactive  and  timely  with  their  input  into  the  test  planning  and 
development  effort. 

The  program  manager  representative  described  the  interaction  between  his 
office  and  the  other  agencies  as  adequate  and  recognized  each  of  the  other  agencies  as 
having  an  important  role  to  play  in  not  only  testing  but  the  entire  development  process. 

b.  Tester 

This  tester  considered  the  scheduling  of  DT&E  the  primary  problem.  He 
credited  PM  over  optimism  and  the  budget/funding  process  as  the  causes.  He  stated  that  by 
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the  time  a  system  is  supposed  to  enter  DT&E,  events  have  ocxurred  that  have  caused  the 
test  window  to  be  reduced  and  or  delayed.  The  tester  may  often  be  left  to  prioritize  the 
van ‘US  tests,  getting  as  many  in  as  possible  (but  not  all)  before  a  report  is  required  and 
decisions  need  to  be  made. 

This  tester  asserted  that  it  was  the  tester’s  job  to  deal  with  the  situation  as  best 
they  can.  If  anything  could  be  done  about  this  it  would  probably  be  the  PM  being  a  little 
more  realistic  about  when  DT&E  will  occur  and  how  long  it  will  take.  The  tester  also 
stated  that  despite  this  and  some  other  problems  "  overall,  the  system  works,  especially  for 
full  scale  development  systems."  [Ref.  451 

The  tester  for  "system  t"  believed  there  was  a  good  relationship  among  the 
agencies  and  a  "fantastic  working  relationship”  between  his  office  and  that  of  the  PM.  He 
attributed  the  system's  overall  success  to  this  interface. 
e,  AMSAA 

The  analyst  saw  scheduling  as  the  primary  problem  with  DT&E.  Specifically 
he  noted  that  on  this  program  as  well  as  others  there  was  usually  not  enough  time  to  test, 
collect  data,  compile  data,  analyze  data,  make  preliminary  reports  and  briefings  and  Hnal 
reports  and  briefings.  His  analysis  team  often  found  itself  working  with  incomplete  data 
sets  from  which  they  were  expected  to  make  final  reports  required  by  the  schedule.  His 
primary  concern  was  that  key  decisions  were  often  made  prior  to  completion  of  flnal 
reports. 

To  help  alleviate  this  problem  he  believed  that  AMSAA  should  be  more 
realistic  when  signing  up  to  a  schedule  and  find  ways  of  reducing  their  internal  processes. 
He  said  that  his  agency  was  addressing  the  problem  by  working  on  mediods  to  shorten 
their  response  time.  He  thought  that  die  best  way  for  t’ le  PMO  to  deal  with  this  issue  was 
to  consider  all  that  is  involved  in  the  evaluation  portion  of  Test  and  Evaluation.  He  said  the 
testers  do  the  best  they  can  given  the  environment  in  which  they  must  test. 
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The  lead  analyst  for  "systenj  t"  considered  the  interface  between  his  office,  the 
tester  and  PMO  as  good.  There  were  regular  TIWG  meetings,  monthly  reviews,  ad  hoc 
meetings  and  special  working  groups.  He  said  there  was  little  interface  between  his  office 
and  those  of  user  and  the  contractor  but  did  not  see  this  as  an  issue. 

d.  User  Representative 

The  user  representative  for  "system  t"  indicated  that  the  most  significant 
problem  with  DT&E  was  that  the  PM  pushed  the  schedule  rather  than  product  readiness. 
He  believed  this  problem  was  a  result  of  the  funding  and  budget  cycles.  The  consequences 
of  this  schedule  push  was  an  early  "bad  name"  for  the  system.  He  said  that  this  system 
initially  received  a  bad  reputation  because  it  went  into  a  testing  functionally  unprepared  for 
the  test. 

The  user  representative  thought  that  his  office  should  have  been  more 
involved  in  the  product  design.  He  believed  that  the  contractor  and  his  office  needed  a 
closer  and  earlier  interface.  He  indicated  that  the  contractor's  engineers  still  do  not 
understand  the  "real"  operational  environment 

The  user  representative  for  this  program  interacts  with  the  other  agencies 
through  action  officers  and  through  the  PMO.  The  user  representative  believed  that  for  this 
system  the  interface  along  with  the  TTWG  is  normally  adequate  but  recommends  more 
frequent  TIWGs  and  a  more  definitive  TIWG  schedule. 

e.  Contractor 

The  contractor  believed  obtaining  the  necessary  resources  to  conduct  a 
successful  DT&E  was  the  major  problem  in  entering  DT&E.  Namely,  needed  activities 
were  delayed,  not  accomplished  or  shortcuts  taken.  Also,  resources  such  as  funding  were 
right  and  schedules  were  compressed  to  the  point  that  completing  the  test  by  a  specific  date 
became  more  important  than  conducting  the  test  according  to  the  test  plan.  Subsequently 
tests  became  more  difficult  to  conduct  and  results  harder  to  understand.  The  contractor 
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stated  these  types  of  practices  early  in  the  program  development  tended  to  push  problems 
into  later  phases  where  they  were  more  expensive  in  terms  of  both  cost  and  schedule. 

The  contractor  thought  that  all  the  agencies  involved  could  help  alleviate  this 
problem.  All  agencies  should  provide  better  estimates  of  resources  in  terms  of  time  and 
money.  The  PM  should  allow  for  contingencies  and  not  assume  perfect  and  complete 
success  at  every  step  along  the  way.  The  contractor  stated  that  long  term  stable  funding 
would  be  the  greatest  help  in  overcoming  schedule  and  resource  problems,  but  that  means 
legislative  action. 

The  prime  contractor  for  "system  t"  was  very  positive  about  the 
contractor/Govemment  PM  relationship  "for  this  program."  He  made  the  point  that  his 
experience  with  other  programs  between  his  company  and  the  Army  were  not  as 
cooperative.  His  interaction  with  the  other  agencies  was  mostly  through  the  TIWG 
process.  The  contractor  thought  for  this  program  the  working  relationships  were  good  and 
provided  an  "easy  flow"  of  information  and  good  cooperation. 

2.  System  u 

a.  Program  Management  Office 

The  PMO  representative  said  the  most  significant  problem  was  that  technical 
tests  were  conducted  before  the  PMO  and  the  contractor  had  sufficient  time  to  do  their  own 
"checking  out"  of  the  system.  This  led  to  surprises  during  DT&E  impacting  test  schedule 
and  costs.  The  PM  representative  believed  this  happened  because  of  inflexible  budgets, 
changing  Operational  Requirements  Documents  (ORDs)  and  because  the  PM  was  locked 
into  an  inflexible  success  oriented  schedule. 

He  suggested  that  the  PM,  through  management,  should  be  able  to  work  these 
issues.  First,  the  PMO  should  get  all  the  players  involved  early,  including  testers  and 
evaluators,  and  concentrate  on  making  realistic  estimates.  Then,  the  PM  needs  to  ensure 
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that  the  ORD  remains  solid  and  realistic.  He  also  said  that  PMs  should  view  the  tester  as 
part  of  the  team  and  not  the  "bad  news  messenger." 

The  Program  Management  Office  from  "system  u"  was  very  pleased  with  the 
working  relationships  that  were  established  within  the  program.  Besides  the  formal  TIWG 
interface,  the  PMO  had  established  other  semi-formal  groups  to  address  numerous  issues 
including  testing.  The  PM  representative  believed  these  working  groups  and  their  efforts 
enhanced  the  TIWG  meetings  as  well  as  other  aspects  of  the  program. 

b.  Tester 

This  tester  believed  that  the  schedule  was  a  major  DT&E  problem  for  this 
program.  He  said  early  involvement  from  the  tester  and  the  analysts  is  key  to  helping 
minimize  this  problem.  The  tester  suggested  that  the  PMO  ensure  testers  and  analysts  are 
involved  early  on  and  that  they  actively  scrutinize  test  and  evaluation  schedules  before  they 
are  finalized.  The  tester  for  "system  u"  was  satisfied  with  the  interface  and  coordination 
that  occurred  for  this  program. 

c.  AMSAA 

The  AMSAA  representative  for  "system  u"  identified  the  changing  of  software 
/hardware  rr-  Irements  as  the  biggest  problem  or  issue  in  entering  Developmental  Test. 
Requirement  changes  often  occurred  after  estimates  were  made  and  therefore  affected  the 
test  plans  and  analysis.  This  changing  test  environment  reduced  confidence  in  the  tests  and 
impeded  analysis. 

The  AMSAA  representative  indicated  that  his  agency  can  help  with  this 
problem  by  working  ’""'h  the  PMO  to  help  identify  problems  early  in  development.  He 
also  stated  that  AMSAA,  the  testers  and  the  contractors  must  do  a  better  job  cf  controlling 
test  costs. 

The  analyi  ^  "system  u"  said  the  interface  between  his  office  and  that  of  the 
PM  started  as  adversarial  but  improved  over  time.  He  believed  he  had  a  good  working 
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relationship  with  the  tester  and  with  the  contractor.  The  analyst  was  not  satisfied  with  the 
contact  that  he  had  with  the  user  representative  and  believed  that  the  analysis  people  should 
obtain  more  feedback  from  the  troops. 

d.  V&er  Representative 

This  user  representative  thought  that  the  major  problem  in  entering  a  DT&E 
was  the  flow  of  critical  paper  work.  Namely,  he  pointed  out  those  documents  (TEMP, 
Detailed  Test  Plan)  which  are  needed  to  make  things  happen.  When  these  documents  are 
late,  it  impacts  and  often  delays  the  test  scliedule.  The  user  representative  believed  for  their 
part  they  should  concentrate  more  on  the  war  fighting  capabilities  and  enhancements  rather 
than  technical  issues.  This  may  help  reduce  the  paper  delays. 

The  user  representative  stated  that  his  contact  with  the  various  agencies  was 
frequent  and  provided  for  a  good  flow  of  information.  The  only  interface  that  did  not  have 
regular  communication  was  that  with  AMSAA.  There  was  only  minimal  contact  with 
AMSAA  and  it  was  usually  formal  in  nature.  He  thought  the  TIWGs  provided  an  adequate 
single  forum  to  bring  the  key  players  together. 

e.  Contractor 

The  contractor  cited  unanticipated  problems  that  delayed  test  completion 
within  schedule  and  increased  test  costs  as  the  major  issues  for  DT&E.  He  ascertained  that 
these  occur  because  proposed  estimates  for  cost  and  schedule  usually  assume  no  technical 
difficulties  will  be  encountered.  Subsequently  overly  optimistic  estimates  become  the 
standard  to  meet. 

This  contractor  representative  saw  ways  for  his  office,  the  PM  and  the  Tester 
to  address  this  issue.  One  way  in  which  the  contractor  believed  that  his  company  could 
help  deal  with  this  problem  was  to  use  actual  schedule  information  from  historical  records 
to  create  more  accurate  future  test  estimates.  He  said  the  tester  also  needs  to  track  test 
program  costs  and  schedule  variances  and  document  these  historical  data  so  that  it  can  be 
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used  when  estimating  future  testing.  He  said  the  PM  should  identify  potential  problems  to 
be  encountered  during  testing  and  formulate  contingency  plans  with  cost  and  schedule 
impaa  acknowledged  in  the  original  estimates. 

The  prime  contractor  for  "system  u"  noted  that  the  contract  office  and  the 
Government  PMO  had  a  very  good  working  relationship.  The  interface  with  other  agencies 
like  AMSAA  and  the  test  facility  was  more  formal,  less  frequent,  but  sufficient. 
Concerning  DT&E,  he  does  not  recall  involvement  with  the  Combat  Developer. 

3.  System  v 

a.  Program  Management  Office 

The  program  representative  for  "system  v,"  an  NDI  Program,  identified  the 
primary  problem  in  conducting  DT&E  as  the  belief  that  the  PM  has  plenty  of  money  for 
test.  He  stated  that  testers  and  analysts  tended  to  want  to  test  extensively  to  reduce  their 
risk  and  increase  their  confidence.  However,  the  PM,  like  all  PMs,  had  a  limited  )udl;;tet 
and  it  was  for  more  than  testing  alone.  He  attributed  the  problem  to  the  acquisition  process 
and  the  congressional  funding  system. 

Because  the  PM  representative  determined  the  root  of  the  problem  to  be  the 
acquisition  process,  he  suggested  that  the  problem  was  beyond  the  scope  of  the  agencies 
and  offices  targeted  for  research.  He  did  suggest  that  the  PM  bring  in  all  the  key  players 
early,  particularly  AMSAA. 

The  PM  representative  described  his  working  relationship  with  the  other 
agencies  as  regular  and  productive.  Most  of  the  interface  is  formal  but  gets  more  familiar 
and  frequent  as  major  tests  or  milestone  reviews  approach. 

b.  Tester 

The  tester  indicated  that  the  primary  problem  in  conducting  Developmental 
Testing  was  reactive  involvement  by  the  PMO  instead  of  proactive  involvement  The  tester 
believed  that  the  PMO  and  the  contractor  often  saw  DT&E  as  an  area  to  maybe  save  some 


time  or  funds.  The  PM  failed  to  put  emphasis  on  DT&E  until  after  something  went  wrong 
and  both  cost  and  schedule  were  negatively  impacted. 

The  tester  believed  that  the  test  community  as  a  whole  should  educate  PMs  to 
the  various  test  capabilities  available.  Also,  testers  should  demonstrate  that  the  test  facilities 
are  more  flexible  than  ever  in  packaging  programs  for  the  PM.  He  stated  the  main  action  a 
PM  can  do  to  avoid  such  a  problem  is  to  get  the  tester  and  the  analysts  involved  early.  The 
analysts  for  their  part  need  to  become  flexible  in  packaging  the  analysis  and  evaluation. 
Tlie  analysts  (AMS  AA)  must  also  realize  that  money  no  longer  exists  for  huge  sample  sizes 
and  that  other  methods  are  needed  to  analyze  and  design  tests. 

The  tester  for  "system  v"  believed  his  interface  with  the  other  agencies  was 
adequate  and  was  particularly  good  with  AMSAA.  Face  to  face  coordination  is  easily 
achieved  due  to  the  close  proximity  to  most  of  those  agencies.  He  stated  that  good 
communication  between  the  tester,  the  PM  and  AMSAA  is  important  for  successful  DT&E. 
c.  AMSAA 

The  AMSAA  representative  for  "system  v"  described  the  primary  issue  in 
DT&E  as  the  Non-Developmental  Item  (NDI)  status  of  the  system.  It  was  assumed  for 
NDI  programs,  because  they  are  "non  developmental,"  that  testing  and  analysis  would  be 
faster  and  easier.  However,  any  type  of  a  problem  during  DT&E  or  any  other  area  in  such 
a  program  can  be  costly.  Problems  in  DT&E  brought  public  scrutiny  and  possibly 
jeopardized  the  entire  system  development.  This  system,  although  NDI,  still  required 
extensive  testing  and  data  collection.  The  task  of  data  reduction,  analysis,  and  report 
preparation  was  reduced  to  a  shorter  time  ftame  because  of  the  NDI  status. 

The  analyst  suggested  some  things  that  could  be  done  by  the  various  players 
to  mitigate  tliis  problem.  The  AMSAA  representative  said  that  AMSAA  is  using  different 
methods  of  analysis  and  evaluation  to  try  to  speed  up  the  evaluation  process.  These 
methods  include  the  physics  of  failure  and  the  reliability  growth  model.  The  physics  of 
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failure  is  the  method  of  using  more  current  electronic  failure  analysis  instead  of  standards 
and  specUHcations  derived  from  early  electronic  hardware.  The  reliability  growth  model 
tracks  the  increasing  reliability  of  a  system  through  its  development  and  projects  and  plans 
for  levels  of  reliability  at  system  maturity.  The  analyst  stated  that  the  PM  needs  to 
recognize  how  non  developmental  a  system  truly  is  and  develop  realistic  schedules.  The 
PM  should  also  ensure  that  NDI  is  not  automatically  associated  with  easier  testing  and 
evaluation.  Finally,  the  analyst  thought  that  the  NDI  test  environment  requires  AMSAA, 
the  tester,  the  PM  and  the  contractor  to  have  a  "team"  approach  to  the  test. 

The  analyst  from  AMSAA  for  "system  v"  was  very  pleased  with  the  working 
relationships  that  had  been  established  among  the  various  agencies.  The  communication 
with  both  the  test  facility  and  the  PMO  was  positive  and  frequent.  There  was  little 
interaction  with  the  user  representative  and  the  contact  with  the  contractor  was  limited  to 
formal  forums  such  as  scoring  conferences  or  TIWGs. 
d.  User  Representative 

The  Combat  £)eveloper  addressed  the  changing  of  program  timeline  as  the 
major  problem  on  entering  a  Development  Test.  Specifically  he  refened  to  the  compressing 
of  the  DT&E  schedule  aitd  an  unplanned  DT&E  and  OT&E  overlap  in  order  to  make  up  lost 
schedule  time.  For  example,  the  PM  and  the  contractor  prepared  for  OT  I,  conducted  DT I 
during  the  preparation  and  tried  to  correct  DT  I  deficiencies  before  the  start  of  OT  I.  The 
Combat  Development  Office  believed  that  this  problem  "was  driven  by  the  desire  to  always 
present  the  program  in  a  positive  light  (otherwise  risk  f  unding  cuts)."  [tlef.  46] 

The  user  representative  office  suggested  the  best  way  to  deal  with  the  issue 
internally  was  to  stake  out  a  performance  issue  such  as  reliability  and  stick  to  it.  This  gives 
the  PM  at  least  one  solid  perspective  as  to  the  system's  readiness  for  test  The  user 
representative  also  said  the  PM  should  realistically  assess  performance  and  system 
readiness  based  on  the  user's  requirements  and  the  system’s  performance,  not  on  "what  it 
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will  take  to  get  to  the  next  hurdle.”  As  for  the  other  agencies  and  personnel  involved,  the 
user  suggested  that  they  need  to  be  prepared  to  make  the  tough  decisions,  e.g.  stop  and  fix 
the  test  process  if  needed. 

The  user  for  "system  v”  said  the  interface  with  the  other  agencies  is  generally 
adequate  but  requires  more  intense  coordination.  He  stated  that  the  various  conferences 
and  working  groups  both  fonnal  and  infoimal  nonnally  achieve  their  intent  but  often  fail  to 
resolve  major  conflicts. 

e.  Contractor 

The  contractor  considered  the  biggest  problem  or  issue  in  entering 
Development  Testing  the  question  of  "how  well  a  system  made  of  .nany  conunercial  parts 
will  perform  under  rigorous  military  testing?"  The  problem  stems  from  the  Government's 
desire  to  have  non-developmental  systems  yet  maintain  military  specifications  and 
standards.  In  some  cases  the  commercial  parts  cannot  hold  up  to  the  stringent  military  tests 
and  standards. 

The  contractor  suggested  that  everyone,  especially  the  PM  should  be  more 
aware  of  the  complexities  in  buying  commercial  items  for  military  use.  Tradeoffs  have  to 
be  made  when  buying  under  the  commercial  use  concept  (time  versus  cost  versus 
performance).  He  believed  contractors  should  challenge  the  various  military  specifications 
and  ascertain  if  a  lesser  performance  level  would  be  acceptable.  The  tester,  who  has 
knowledge  and  experience  should  be  involved  in  contract  speciHcations  review  for  (NDl) 
contracts.  Finally,  the  contractor  recommended  that  the  Combat  Developer  should  help 
assess  tradeoffs  and  delete  unnecersaiy  requirements. 

The  Contractor  for  "system  v"  described  different  levels  of  interaction  among 
the  agencies.  There  was  regular  communication  between  the  contractor's  program  office 
and  the  Government  program  office.  There  was  also  regular  interface  between  the 
contractor  and  both  AMSAA  and  the  tester.  The  contractor  had  support  personnel  at  two 
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test  facilities  attend  meetings  and  facilitate  good  communications.  The  Interface  with  the 
Combat  Developer  was  less  frequent  but  increased  as  the  system  prepared  for  another  test. 

4.  System  w 

a.  Program  ManagomtM  Office 

The  program  representative  stated  dtat  his  major  problem  in  conducting  DT&E 
was  that  you  test  regardless  of  how  ready  you  are,  "...it's  a  mark  on  the  board  you  must 
meet."  The  office  also  said  there  needs  to  be  more  control  over  the  testing.  Some  tests  are 
conducted  to  satisfy  an  evaluator's  need  but  may  add  little  to  the  overall  analysis  of  the 
system.  AMSAA  particularly  is  not  required  to  test  prudently.  All  this  non  value  added 
testing  just  makes  completing  DT&E  within  a  very  optimistic  schedule  that  much  harder. 

The  Program  representative  suggested  some  actions  for  the  various  offices  to 
deal  with  these  problems.  First,  the  PM  should  have  everyone  involved  early  -  about  the 
time  of  the  Statement  of  Work  (SOW).  The  test  and  evaluation  participants  need  to  reduce 
and  justify  tests,  and  budgets  accordingly.  He  also  stated  that  test  facilities  have  recently 
become  aware  of  this  situation  and  have  responded,  but  the  evaluators  were  not  as 
responsive.  ThePMO  believed  the  Combat  Developer  should  play  a  more  definitive  role 
in  testing.  A  strong  combat  development  office,  that  knows  the  system’s  background  and 
the  doctrine,  could  help  decrease  testing  that  adds  no  value  to  the  system. 

The  program  office  responded  that  they  had  a  good  interface  with  the  other 
agencies.  They  believed  the  formal  interface  of  the  TIWG  was  good  but  attributed  the 
positive  working  relationship  between  agencies  to  die  fact  that  aU  the  agencies  were  brought 
in  early. 

h.  Tester 

The  tester  for  "system  w"  regarded  the  "lack  of  concrete  requirements"  as  the 
most  significant  problem  in  conducting  DT&E.  This  led  to  difficulties  in  test  planning  and 
resulted  in  schedule  and  cost  overruns. 
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The  tester  admitted  that  his  office  could  do  better  at  coordinating  the  test  effort 
with  the  PM,  AMSAA,  and  the  Combat  Developer,  but  that  the  PM  is  the  one  who  must 
bring  these  groups  into  harmony  concerning  the  test  requirements.  As  for  the  Combat 
Developer,  the  tester  said  he  needed  to  define  the  requirements,  learn  and  understand  the 
system  and  appreciate  the  impact  that  changing  requirements  have  on  the  test  process. 
AMSAA  should  also  try  to  better  understand  the  system  under  test 

The  tester  was  pleased  with  the  interaction  and  working  relationships  with  the 
other  agencies.  The  one  exception  he  noted  was  that  his  interface  with  the  Combat 
Developer  was  limited  to  the  formal  meetings  such  as  TIWGs.  He  said  that  the  working 
relationship  was  good  but  needed  to  be  more  frequent 
c.  AMSAA 

The  analyst  from  AMSAA  indicated  that  the  biggest  problem  with  DT&E  was 
its  uncertainty.  That  is.  did  the  tester  have  adequate  control  to  complete  all  testing  on 
schedule  and  within  budget?  He  further  explained  that  some  tests  were  not  performed  due 
to  lack  of  time,  funding  or  both.  This  creates  "data  voids"  and  makes  the  evaluation  more 
difficult 

The  AMSAA  representative  believed  that  all  the  agencies  can  help  at  least 
mitigate  the  problem,  but  also  asserted  that  it  will  take  legislative  action  to  get  testing  "event 
driven  rather  than  schedule  driven."  He  stated  for  AMSAA's  part,  they  should  prioritize 
testing  and  work  closely  with  both  the  PM  and  tester  to  design  tests  that  ut  within  the  given 
schedule  and  budget.  The  analyst  also  suggested  that  the  PM  fund  the  tester  "as  needed" 
rather  than  by  fiscal  year.  The  tester  must  learn  more  about  the  system  under  test  and  try  to 
anticipate  potential  test  control  problems.  The  contractor  should  work  closer  with  the  tester 
to  integrate  the  test  item  and  the  instrumentation.  Finally,  the  analyst  said  that  the  Combat 
Developer  should  provide  more  explicit  guidance  on  test  set  up  and  installation  procedures. 


The  AMSAA  representative  concluded  that  the  interface  with  the  other 
agencies  was  normal  and  in  most  cases  sufficient.  The  TIWG,  the  most  common  forum, 
was  adequate  for  keeping  the  test  community  abreast  of  the  status  and  latest  developments 
in  the  progxam,  but  lacked  in  solving  detailed,  lower  level  issues. 
d.  Contractor 

The  contractor  for  "system  w"  said  that  the  biggest  problem  in  conducting 
DT&E  was  the  lack  of  adequate  coordination  between  the  tester  and  the  contractor 
especially  when  integration  checkout  was  needed  between  the  test  instrumentation  and  the 
system  under  test.  This  can  .severely  impact  both  schedule  and  cost  if  restarts  and  retests 
are  needed. 

The  contractor  indicated  that  he  is  limited  to  bringing  the  issue  to  the  attention 
of  the  PM  and  explaining  its  potential  effect  on  the  test  and  test  data.  He  believed  closer 
coordination  with  the  tester  and  a  better  technical  understanding  by  the  tester  would  help 
alleviate  the  problem,  but  the  PM  needs  to  influence  such  a  reladonsiiip.  He  also  suggested 
that  the  Combat  Developer  establish  realistic  and  unchanging  requirements  that  will  give  the 
other  agencies  a  foundation  to  develop  tests  and  evaluation  plans. 

The  contractor  described  his  interface  with  the  other  agencies  as  regular  and 
sufficient.  Contact  with  the  PM  was  both  formal  and  informal  and  contact  with  AMSAA 
and  the  tester  usually  in  relation  to  reviews  or  technical  documentation.  He  said  there  was 
very  little  contact  with  the  Combat  Developer. 

5.  System  x 

a.  Program  Management  Office 

The  Program  Management  Office  stated  that  "The  major  problem  entering  this 
developmental  test  program  was  the  compressed  test  schedule."  The  test  schedule  was  laid 
out  with  no  flexibility  and  testing  continued  throughout  the  program.  When  problems  with 
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design  or  other  areas  were  encountered,  delays  and  slips  occurred.  The  slips  in  turn 
delayed  and  or  compressed  DT&E  as  well  as  affected  other  areas  such  as  OT&E. 

The  Program  Management  Office  had  recommendations  for  the  various 
agencies  in  addressing  this  problem.  For  the  Program  Management  Office  itself,  the 
representative  believed  the  PM  must  recognize  the  importance  of  the  test  schedule  early  and 
make  a  concerted  effort  to  hold  the  line  on  design  reviews,  hardware  deliveries  and  costs. 
The  PM  representative  saw  the  tester  as  being  left  to  complete  testing  within  many 
constraints.  He  suggested  that  the  tester  try  to  be  innovative  and  find  ways  to  expedite 
testing.  He  believed  the  contractor  could  help  by  delivering  hardware  on  time  and 
committing  sufficient  resources  to  support  the  test.  The  PM  representative  suggested 
AMSAA  should  be  open  minded  to  problems,  potential  solutions  and  be  able  to  make  quick 
decisions.  The  Combat  Developer  also  needs  to  respond  to  questions  and  issues  in  a  timely 
manner. 

The  PMO  for  "system  x"  characterized  the  working  relationship  with  the  other 
agencies  as  frequent,  direct  and  both  formal  and  informal.  The  PM  representative 
considered  this  interface  sufficient 
c.  AMSAA 

The  analyst  for  "system  x"  indicated  limited  sample  size  as  the  major  problem 
in  developmental  testing  and  evaluation.  A  limited  sample  size  causes  data  analysis  to  be 
more  difficult  and  increases  risk.  The  analyst  recognized  that  smaller  sample  sizes  were 
becoming  the  norm  as  funding  continues  to  decrease. 

He  recommended  that  AMSAA  rely  more  heavily  on  models  and  take  part  in 
building  reliable  models.  AMSAA  also  should  consider  modeling  and  simulation  in  test 
design. 
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6.  System  y 

ff.  Program  Management  Office 

The  Program  Management  Representative  for  "system  y,"  an  NDl  program, 
stated  that  people  have  great  expectations  of  NDI  programs  and  so  the  testing  schedule  for 
such  a  program  is  very  intensive.  Trying  to  fit  in  all  the  testing  that  is  required  becomes 
difficult  and  is  the  primary  DT&E  issue  for  diis  type  of  program. 

The  PM  representative  for  this  system  believed  NDI  type  programs  will 
continue  to  be  tested  in  an  accelerated  fashion  and  suggested  ways  that  the  PM  and  other 
agencies  could  deal  with  this  issue.  He  asserted  that  the  PM  should  hold  "conclusive" 
TIWGs.  That  is,  "Make  the  TIWG  important  and  ensure  that  the  other  agencies  send 
representatives  that  can  make  decisions  and  can  speak  for  that  office."  [Ref.  47]  The  PM 
also  should  have  the  tester  and  analyst  on  board  early.  He  indicated  that  the  tester  should 
coordinate  the  test  effort  and  start  as  early  as  possible.  The  analysts  should  accept  some 
risk  and  the  Combat  Developer  should  develop  a  good  set  of  requirements  and  then  stick  to 
them.  Changing  requirements  severely  impact  the  already  intensive  test  schedule.  The 
contractor  should  dedicate  the  right  people  to  the  test  and  concentrate  on  putting  them  at  the 
right  place  during  the  key  test  events. 

This  person  indicated  that  the  interface  with  the  other  agencies  was  good,  but 
that  the  PM  could  improve  the  results  through  better  manage.nent  of  the  TIWG  process  as 
previously  cited. 

b.  AMSAA 

The  analyst  considered  the  use  of  contractor  test  plans  and  test  data  for 
evaluation  and  analysis  as  the  major  issue.  Due  to  shrinking  Government  resources,  the 
use  of  contractor  data  is  becoming  more  common. 

To  address  this  problem  the  analyst  suggested  that  AMSAA  should  be  more 
vigilant  in  reviewing  test  information  provided  by  the  contractor  and  the  contractor  should 
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be  more  receptive  to  some  of  the  unique  oversight  required  by  the  Government.  The 
analyst  also  stated  that  the  PMO  should  build  safeguards  into  the  TEMP  to  deal  with  using 
contractor  testing  and  data.  Finally  the  analyst  concluded  that  even  if  the  Government  does 
not  conduct  the  test  in  a  certain  case,  the  expertise  of  the  tester  will  still  be  needed  as  will 
some  of  the  Government  test  facilities. 

The  analyst  for  this  system  believed  that  the  working  relationship  between  her 
office  and  that  of  the  other  agencies  was  sufficient.  Hie  contact  between  the  various 
groups  was  conducted  through  both  formal  and  informal  means  and  conducted  frequently. 
The  analyst  noted  that  the  proximity  of  AMSAA  to  TECOM  Headquarters  was  a  positive 
contributor  to  the  good  exchange  of  information. 

c .  User  Representative 

The  Combat  Developer  for  this  system  considered  the  decreasing  funds  for 
RDT&E  as  the  most  significant  problem  in  conducting  DT&E  as  well  as  other  types  of 
testing.  The  decreased  funding  has  caused  compromised  and  reduced  testing  and  increased 
risk.  This  reduced  T&E  and  increased  risk  environment  could  be  acceptable.  However, 
typically  when  problems  occur  with  a  system,  tiie  program  suddenly  becomes  a  target  for 
inquiry,  funding  reductions  or  even  elimination  because  ".  .the  Army  failed  to  properly 
test."  [Ref.  48] 

The  Combat  Developer  conceded  that  decreasing  funding  is  a  reality  and 
believed  it  will  continue  for  the  next  few  years.  The  best  way  that  his  office  can  deal  with 
this  problem  is  to  actively  participate  in  the  HWG  process  and  insure  the  TEMP  suppons 
the  ORD  and  the  tequirem  ;,ib>  in  the  ORD  are  valid  and  realistic. 

d.  Contractor 

The  contractor  for  "system  y"  stated  that  the  major  problem  in  conducting 
DT&E  was  the  inconsistency  between  the  ORD  and  the  contract  requirements.  The 
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inconsistencies  increased  test  cost  and  test  time.  Schedule  was  regained  but  at  additional 
cost  to  both  the  Government  and  the  contractor. 

The  contractor  suggested  some  actions  to  be  taken  if  the  inconsistency  occurs 
and  also  actions  to  prevent  the  problem  in  the  first  place.  First,  in  order  to  resolve  an 
existing  problem,  the  TIWG  members  did  an  extensive  cross  match  of  the  contract,  the 
ORD  and  flic  TEMP.  This  created  an  overall,  though  not  complete,  consensus  among  the 
TIWG  members.  Additionally,  continuous  tracking  of  the  test  issues  by  the  PM  helped 
resolve  the  problems.  To  avoid  variation  between  the  ORD,  the  contract,  and  the  TEMP  in 
the  first  place  the  contractor  suggested  that  the  PM  and  the  Combat  Developer  thoroughly 
review  the  contract  and  verify  that  it  corresponds  to  the  ORD. 

The  cono  actor  described  the  relationships  between  his  office  and  the  various 
agencies  as  both  formal  and  informal  and  as  sufficient.  He  further  stated  that  he  did  not 
believe  the  program  testing  would  have  gone  as  well  if  they  had  strictly  relied  on  the  formal 
interface  of  the  TIWG  alone.  The  informal  working  relationships  were  a  key  to  the  overaU 
test  success. 

7.  System  i 

a.  Program  Management  Office 

The  Program  Management  Office  representative  for  this  program  cited  the 
Government's  changing  requirements  as  the  biggest  problem  in  conducting  a  DT&E. 
Changing  requirements  are  difficult  for  any  program  test,  but  with  software  intensive 
programs  it’s  "really  tough."  [Ref.  49]  When  unplanned  and  unfounded  requirements  keep 
coming,  none  of  the  documentation  is  solidified.  The  detailed  test  plan,  the  software  test 
plan,  the  Independent  Evaluation  Plan  (lEP)  and  the  TEMP  are  all  affected  by  the  changing 
requirements  as  is  the  test  schedule. 

The  PM  representative  saw  some  ways  to  address  this  problem.  First,  he  said 
that  the  tester  had  to  be  involved  early  and  be  kept  up  to  date  on  system  changes  that  could 
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affect  the  test.  Next,  the  tester  should  realize  the  complexities  of  software  testing.  Finally, 
the  various  agencies  including  the  contractor  need  to  make  a  team  effort.  He  thought  for 
"system  z"  that  AMSAA  took  on  an  antagonistic  role  and  that  the  contractor  was  not  up 
front  with  bad  news.  A  "team  member  spirit"  might  have  helped  avoided  these  problems. 

b.  AMSAA 

The  AMSAA  spokesperson  for  this  system  believed  that  the  major  problem  in 
conducting  DT&E  was  that  DT&E  was  a  target  for  cutting  costs.  He  suggested  that  this 
occurs  because  PMs  are  cost  and  schedule  driven  and  that  by  the  time  DT&E  rolls  around 
many  programs  have  cost  and  schedule  overruns.  PMs  start  looking  for  ways  to  save  and 
they  cut  out  some  development  type  tests.  Cutting  tests  impacts  data,  data  analysis  and 
creates  more  risk  and  development  uncertainty. 

He  recommended  that  the  PM  and  the  contractor  should  "realize  the  value  of 
DT&E."  They  needed  to  "accept  testing  instead  of  seeing  it  as  a  burden."  He  also  said  that 
the  contractor  should  be  up  front  with  potential  problems,  especially  software  problems. 

He  described  the  interface  between  his  office  and  the  other  agencies  as 
changing.  Originally  it  was  forma!,  mostly  TIWGs  and  teleconferencing,  but  this  has 
improved  by  becoming  more  frequent  and  including  other  forums  and  less  formal  working 
relationships. 

c.  User  Representative 

The  user  representative  for  "system  z"  regarded  the  lack  of  regulation  and 
guidance  for  software  testing  to  be  the  primary  problem  in  conducting  DT&E.  This  made 
the  development  and  fielding  of  multiple  versions  of  software  extremely  difficult. 

To  resolve  this  problem  the  Combat  Developer  suggested  that  the  PM  office 
push  for  quicker  fielding  decisions,  and  the  testers  and  analysts  lobby  for  changes  to  the 
regulation  and  process  for  testing  software. 
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The  Combat  Developer  indicated  that  the  interface  between  his  office  and  the 
others  was  generally  sufficient  and  that  the  TIWG  was  an  adequate  forum  which  served  the 
purpose  for  which  it  was  designed.  He  noted  that  the  geographical  location  between  the 
agencies;  particularly  between  his  office  and  the  former  contractor  made  that  working 
relationship  insufficient. 

8.  Other  Interviews 

A  number  of  interviews  were  conducted  with  people  who  through  their  position 
and  experience  provided  insight  to  the  thesis  questions.  These  people  represented 
supervisors,  branch  and  division  chiefs  at  a  TECOM  test  facility,  TECOM  Headquarters 
itself  and  AMS  A  A.  These  interviews  focused  on  the  following: 

•  What  do  you  consider  the  primary  problem  or  issue  in  Developmental  Testing? 

•  What  can  your  agency  or  any  of  the  others  you  work  with  do  about  the  problem? 

a.  TECOM  I 

This  project  engineer  said  that  schedule  compression  was  the  major  problem 
in  entering  or  conducting  DT&E.  He  referred  to  the  continuous  pressure  to  conduct 
Developmental  Testing  in  considerably  less  time  than  originally  estimated.  He  believed  that 
the  funding  and  budget  process  drove  this  compressed  schedule  and  caused  major 
decisions  to  be  made  with  only  partial  data.  He  also  said  that  the  budget  process  caused  the 
PM  to  focus  on  budget  and  Initial  Operational  Capability  (IOC),  therefore  risking  system 
quality.  He  suggested  the  best  way  to  deal  with  this  issue  is  for  the  PM  to  understand  early 
on  what  the  capabilities  are  in  the  test  and  evaluation  community  and  to  work  closely  with 
the  tester  and  analyst. 

b.  TECOM  II 

This  representative  from  TECOM  thought  that  "success  oriented  testing"  was 
the  major  problem  in  entering  or  conducting  DT&E.  This  unrealistic  and  optimistic  attitude 
makes  the  tester  a  "bad  guy  or  gal"  when  a  test  reveals  a  problem  with  the  system.  It  also 
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fails  to  allow  the  contractor  the  time  he  should  have  to  improve  his  design.  Because 
problems  are  not  planned  for,  it  negatively  impacts  both  cost  and  schedule  when  they  do 
occur.  PMs,  contractors  and  everyone  else  should  anticipate  and  identify  potential 
problems.  To  alleviate  this  problem  he  thought  that  all  the  agencies  should  simply  be  more 
realistic  about  developing  a  test  schedule. 

c .  AMSAA  I 

This  senior  analyst  said  the  problem  in  entering  developmental  test  was  that  in 
many  cases  we  launch  into  testing  with  hardware  or  software  that  in  reality  is  not  ready  for 
test.  He  believed  that  PMs  do  not  always  receive  a  realistic  test  status  picture  from  their 
staffs.  No  one  wants  to  deal  with  bad  news,  even  potentially  bad  news.  The  analyst 
suggested  that  PMs  insure  that  their  staff  representatives  for  T&E  have  some  test 
experience  and  coordinate  closely  with  the  testers  and  evaluators. 

d.  AMSAA  // 

Another  senior  analyst  from  AMSAA  considered  the  schedule  driven 
environment  versus  an  event  driven  environment  as  the  primary  problem  in  conducting 
developmental  testing.  From  the  analysis  point  of  view  this  causes  problems  with  data 
collection,  analysis  and  reporting.  He  feels  this  problem  is  not  easily  resolved  by  any 
agency  or  even  all  the  agencies,  because  program  funding  is  also  schedule  or  calendar 
driven  and  all  the  participants  know  that  He  suggested  that  early  coordination,  team  work 
and  realistic  estimates  by  all  the  participants  could  minimize  the  effects  of  this  problem. 

e.  AMSAA  III 

The  next  analyst  said  the  most  significant  problem  was  trying  to  obtain  an 
adequate  sample  size,  one  large  enough  to  analyze  and  yet  not  so  large  that  it  is  cost 
prohibitive.  He  believed  his  agency  must  move  to  using  more  simulation  and  modeling  and 
validate  that  information  with  a  small  number  of  actual  tests.  He  also  stated  that  more 
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positive  incentives  for  contractors  would  be  valuable.  Rewarding  the  contractor  early,  for 
good  designs  that  pass  early  basic  testing  would  be  an  excellent  investment 

/.  Tester  / 

This  tester  stated  the  PM  attitude  about  Developmental  Testing  was  the  most 
signiHcant  problem  in  entering  or  conducting  DT&E.  He  said  that  PMs  are  more  worried 
about  the  cost  and  schedule  of  testing  than  on  using  it  as  a  tool.  PMs  often  see  DT&E  as  an 
area  to  try  to  make  up  cost  or  schedule  overruns.  This  attitude  is  created  by  the  process  that 
emphasizes  getting  everything  right  the  first  time.  Testing  often  surfaces  failures  or 
problems  that  the  PM  or  contractor  had  not  anticipated.  Such  failures  can  diastically  impact 
a  success  oriented  schedule. 

The  tester  determined  that  to  improve  this  situation  testers  should  educate  PMs 
as  to  their  testing  capabilities.  Also,  the  decision  makers  need  to  be  realistic  in  their 
expectations  and  let  the  "test,  failure,  fix,  retest"  model  do  its  job. 

g.  Tester  U 

Another  senior  tester  saw  the  lack  of  early  tester  involvement  as  a  primary 
problem  in  conducting  DT&E.  He  believed  even  under  an  unrealistic  schedule,  the  tester, 
if  involved  early  and  kept  informed  could  bring  the  test  resources  needed  for  the  PM's 
requirements.  Early  participation  by  the  tester  and  the  analyst  can  help  the  tester  customize 
the  test  program  to  satisfy  the  various  data  and  analysis  requirements  as  well  as  reduce 
costs.  His  bottom  line  was  "bring  the  testers  into  the  program  early." 

h.  Tester  III 

The  next  tester  thought  that  the  acquisition  process  itself  was  the  major 
problem  in  conducting  DT&E.  The  acquisition  process  causes  PMs  to  be  proponents  of  the 
system  instead  of  proponents  of  the  user.  The  emphasis  is  on  program  success  instead  of 
user  factors.  He  believed  it  is  the  acquisition  process  that  causes  the  unrealistic  schedules, 
the  adversarial  relationships  and  systems  that  are  fielded  needing  modes  and  retrofits  almost 
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immediately.  To  fix  this  problem,  the  decision  makers  at  very  high  levels  (DA,  DOD,  the 
Congress  for  example)  need  to  reward  PMs  that  are  critical  and  objective  about  their 
systems  instead  of  fire  them.  [Ref.  50] 

9.  Recent  Studies  and  Initiatives 

Several  recent  studies  have  made  recommendations  that  apply  to  the  problems 
identified  in  this  thesis.  These  recommendations  include  streamlining  the  T&E  process; 
better  risk  management  and  increased  acceptance  of  risk  at  all  levels;  testing  smarter; 
acquisition  reform;  and  early  user  involvement  in  the  test  process. 

Streamlining  T&E  or  reducing  the  amount  of  actual  testing  and  evaluation  needed 
was  a  common  theme  throughout  the  studies.  In  the  previous  era  (Cold  War),  test  designs 
and  test  plans  called  for  enough  data  generation  to  practically  make  evaluation  a  misnomer. 
Despite  PM  resistance  to  extensive  testing,  it  usually  still  occurred.  In  the  Post  Cold  War 
environment  testers,  evaluators  and  decision  makers  will  no  longer  have  the  luxury  of  an 
unlimited  amount  of  test  data.  [Ref.  51]  Tlic  recommendations  for  reducing  actual  testing 
is  to  use  modeling  and  simulation,  integrate  OT&E  and  DT&E  where  feasible  and  to 
increase  decision  risk  analysis.  The  increase  in  risk  applies  to  testers,  evaluators  as  well  as 
decision  makers.  Instead  of  a  zero  tolerance  mode  for  development  testing,  a  limited 
number  of  test  criteria  should  be  selected  and  an  acceptance  of  test  event  risks  outlined. 
[Ref.  52] 

Throughout  the  studies  and  initiatives,  the  emphasis  was  on  testing  smarter  and 
cheaper.  The  testing  community  in  the  Army;  AMSAA,  TECOM,  OEC,  and  TEXCOM 
recognized  the  need  for  T&E  efficiency  and  reduction  and  have  started  to  formally  meet, 
discuss  and  even  implement  some  of  the  ideas.  One  example  is  the  "improvement  of 
requirements"  determination.  The  emphasis  is  to  develop  ntalistic  requirements  that  meet 
the  user's  needs  and  can  be  efficiently  tes^.  The  following  example  illustrates  the  concept 
of  improvement  of  requirements. 
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A  user  had  mandated  that  a  system  with  a  Mean  Time  Between  Failure  (MTBF)  of 
80  hours  increase  to  a  MTBF  of  ISO  hours  at  system  maturity.  After  evaluating  and 
scrutinizing  the  requirement  it  was  determined  that  a  MTBF  of  ISO  hours  made  no 
significant  impact  on  mission  success  but  increased  program  risk  and  would  take  a  lot 
of  time  to  test.  After  a  realistic  analysis,  the  M^F  was  increased  to  1 14  hours  at 
maturity.  This  took  less  test  dme,  reduced  program  risk  and  improved  the  system 
within  realistic  terms.  [Ref.  S3] 

An  area  mentioned  by  many  respondents  in  the  interviews  was  testing  within  the 
acquisition  process  itself.  The  respondents  normally  suggested  ways  to  resolve  problems 
within  the  system  believing  the  acquisition  process  too  difficult  to  change.  However, 
unlike  other  reform  efforts  in  the  recent  past,  there  is  anticipation  that  an  opportunity  truly 
exists  to  improve  the  process.  The  Cold  War  is  over  and  DOD  resources  are  very  limited 
and  agencies  like  the  GAO  argue  that  this  is  an  ideal  time  to  change  the  system. 

Changes  of  the  type  needed  will  not  come  easily.  They  must  be  directed  at  the 
system  of  incentives  that  has  become  self-sustaining  and  very  difficult  to  uproot.  The 
incentives  that  motivate  the  participants  must  be  realigned  with  better  program 
outcomes.  If  we  expect  program  sponsors  to  be  forthright  about  program 
alternatives,  costs  and  risks,  such  candor  must  be  rewarded,  and  parochialism  and 
undue  optiimsm  penalized.  Ultimately,  change  will  occur  only  through  the  collective 
action  of  the  acquisition  participants,  particularly  within  the  Department  of  Defense 
and  the  Congress,  for  it  is  their  actions  that  dictate  the  incentives  that  drive  the 
process.  [Ref.  54] 

Early  user  involvement  is  another  recommendation  cited  by  the  various  studies. 
The  studies  state  that  the  users  can  help  the  tester  and  evaluator  focus  in  on  those  areas  that 
are  critical  to  mission  success  and  system  performance  rather  than  just  specifications.  Early 
user  involvement  also  should  make  users  more  familiar  with  the  test  environment  so  that 
they  can  help  the  tester  and  other  decision  makers  determine  the  value  and  utility  of  future 
technologies.  It  should  also  give  the  user  an  appreciation  for  the  test  process  and  the 
impact  of  changing  requirements. 

Today  may  in  fact  be  the  best  opportunity  the  Government  has  ever  had  to 
improve  the  overall  acquisition  process.  Such  an  idea  is  politically  popular  and  it  appears 
more  agencies  than  ever  are  looking  into  the  "how"  of  changing  the  process.  Any  changes 
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to  the  acquisition  process  will  likely  affect  testing  and  testing,  in  fact,  continues  to  be  one 
of  the  focus  areas  of  the  present  reform  effort. 
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IV.  RESULTS  AND  ANALYSIS 


A.  INTRODUCTION 

This  chapter  discusses  the  results  and  analysis  of  the  data  presented  in  the  previous 
chapter.  The  focus  for  the  analysis  was  on  the  primary  thesis  question:  "What  do  you 
consider  the  most  significant  problem  in  conducting  Developmental  Testing?"  After 
analyzing  the  agency  responses  it  was  determined  that  five  problems  areas  were  commonly 
noted  across  agencies.  The  responses  were  then  categorized  into  one  of  the  five  common 
problem  areas.  The  categories  included:  (1)  Schedule  problems,  (2)  Problems  with  the 
Acquisition  Process,  (3)  Test  Culture  Problems,  (4)  Resources  Management  and  (S) 
Changes  in  Requirements.  The  problems  were  analyzed  by  system,  by  agency  as  well  as 
by  the  type  of  system  and  its  development  strategy.  This  analysis  led  to  a  unique  finding  in 
reference  to  a  system's  development  strategy  and  discussed  in  the  "Other  Observations" 
section  of  this  chapter..  The  categories  are  explained  below  and  Table  1  presents  a 
simplified  summary  of  the  results. 

1.  Schedule 

This  category  describes  problems  or  issues  related  to  test  schedule  and  insufficient 
test  time.  The  various  respondents  described  these  problems  with  "schedule  crunch  or 
squeeze,"  "lack  of  time  for  proper  testing,"  or  "schedule  push." 

2.  Acquisition  Process 

The  Acquisition  Process  category  describes  the  responses  that  focused  on  testing 
problems  that  are  a  consequence  of  the  overall  acquisition  process.  These  responses 
concentrated  on  the  processes  that  create  problems  for  DT&E  as  well  as  otlier  areas. 
Common  answers  included;  "the  process  creates  unrealistic  optimism  and  expectations," 
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"congressional  funding  does  not  allow  for  good  long  term  planning,"  or  "the  process 
overburdens  both  the  Government  and  the  contractor  with  bureaucracy." 


TABLE  I 

INTERVIEW  RESULTS 


PROBLEM 

AGENCY 

Schedule 

Acquisitiun 

Process 

Test 

Culture 

Resources 

Management 

Change  in 
Requirements 

PMO 

XX 

XXX 

- 

X 

X 

TESTER 

xxxx 

X 

XXXX 

- 

X 

AMS  A  A 

XXX 

XX 

XX 

XX 

X 

CD* 

XX 

XX 

- 

X 

- 

Kr** 

X 

X 

X 

X 

X 

TOTAL 

12 

9 

7 

5 

4 

X  •  A  single  agency  whose  responses  applied  to  that  particular  category. 

*  .  Combat  Developer 

•  •  .  Contractor 


3.  Test  Culture 

The  next  type  of  problem  identified  was  described  as  test  culture.  This  category 
consists  of  the  responses  which  indicated  that  negative  attitudes  and  stereotypes  exist 
toward  testing,  testers  and  analysts.  This  "culture"  is  blamed  for  many  of  the  problems 
including  the  adversarial  relationships  and  inadequate  communication  and  cooperation 
between  the  test  community  and  other  agencies.  Responses  included  comments  such  as 
"late  tester  involvement,"  "lack  of  coordination  due  to  adversarial  role  of  the  tester," 
"tcster/analyst  the  bad  news  messenger,"  "DT  -  a  place  to  make  up  lost  schedule  or  dollars" 
or  "DT  as  an  inconvenivmce  to  the  program ." 
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4.  Resources  Management 

Resource  management  relates  to  problems  noted  by  the  respondents  such  as  lack 
of  funds,  instrumentation,  hardware  or  software.  It  also  refers  to  the  failure  by  one  or 
more  agencies  to  enstire  that  those  same  types  of  resources  are  at  the  right  place,  at  the  right 
time  to  conduct  the  proper  testing. 

5.  Change  In  Requirements 

Change  in  Requirements  is  the  final  problem  area  in  which  responses  were 
categorized.  Problems  of  this  nature  occur  when  changes  in  the  requirements  in  turn 
impact  the  TEMP,  test  conduct  and  or  the  evaluation.  This  problem  was  also  mentioned  by 
many  of  the  respondents,  but  usually  as  an  aside  and  not  as  the  most  significant  problem. 

B.  ANALYSIS 

This  section  summarizes  the  findings  from  the  responses  by  category.  The  categories 
are  further  divided  into  findings  across  systems  and  findings  across  agencies.  Each  of 
these  areas  summarizes  what  the  respondents  determined  as  the  cause(s)  of  the  problem. 
Finally,  for  each  category,  recommendations  arc  provided  for  problem  minimization, 
avoidance  or  prevention. 

1 .  Schedule 

a.  Findings  Across  Systems 

Schedule  was  the  most  fi^uent  problem  mentioned  in  the  interviews.  Five  of 
the  seven  systems  represented  had  at  least  one  respondent  describe  schedule  type  problems. 
In  addition,  of  those  who  stated  that  the  acquisition  process  (next  category)  was  the  major 
problem,  many  pointed  to  the  negative  impact  on  schedule  caused  by  the  acquisition 
process.  The  process  appeared  to  encourage  unrealistic  schedule  estimates  firom  all 
agencies.  The  respondents  generally  concluded  that  extremely  optimistic  management  was 
the  primary  reason  for  schedule  problems.  Schedules  were  developed  months,  even  years 
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in  advance,  always  anticipating  success  along  the  way,  but  tailing  to  take  into  account 
historical  test  information  or  previous  test  experience.  If  the  test  completion  date  could  not 
be  adjusted,  test  reduction  and  compression  resulted  in  order  to  meet  the  schedule. 

b.  Findings  Across  Agencies 

Across  agencies,  at  least  one  representative  from  each  agency  described 
schedule  as  a  significant  problem.  The  testers  and  AMS  A  A  believed  that  schedule 

I 

problems  were  a  product  of  unrealistic  estimates.  Testers  and  analysts  were  either  not 
involved  in  early  estimates  or  they  signed  up  to  an  overly  aggressive  test  schedule.  The 
Combat  Developers  and  PM  offices  cited  over  optimism  and  fear  of  funding  cuts  as  the 
causes  for  schedule  problems.  The  responding  contractor  in  this  category  focused  on 
estimates  that  assume  no  technical  difrlculties  as  cause  of  the  problem. 

c .  Recommendations 

The  following  is  a  summary  of  respondents'  recommendations  to  minimize  or 
prevent  schedule  problems: 

•  PMs  should  push  for  early  involvement  of  all  the  participants  including  the  testers  and 
evaluators. 

•  More  realistic  estimates  should  be  made  by  all  agencies  involved  with  less  optimism 
from  the  PM. 

•  The  PM  should  make  the  testers  and  evaluators  part  of  the  team  and  not  the  bad  news 
messengers. 

•  The  Combat  Developer  should  be  actively  involved  in  test  planning  from  the 
beginning. 

•  Historical  information  and  data  from  previous  tests  should  be  used  to  better  estimate 
future  tests  costs  and  schedule. 

Schedule  was  the  most  common  problem  mentioned  in  tlie  interviews.  The 
respondents  generally  cited  PM  optimism  and  unrealistic  estimates  as  tlie  cause  of  schedule 
problems.  Overall  early  agency  involvement  and  participation  and  realistic  estimates  were 
recognized  as  the  best  methods  to  prevent  or  minimize  schedule  problems. 


2.  Acquisition  Process 

o.  Findings  Across  Systems 

Across  systems,  six  of  the  seven  systems  had  at  least  one  respondent  describe 
the  acquisition  process  as  the  most  significant  problem.  The  major  causes  for  this  problem 
were  the  funding  process  and  PM  over-optimism.  The  annual  control  of  funds  forces  PMs 
to  be  optimistic  and  show  positive  progress  in  cost,  schedule  and  performance  on  a  regular 
basis.  If  not,  the  PM  faces  possible  funding  cuts. 

PM  optimism  forces  other  agencies  to  plan  and  schedule  based  on  the  PM's 
extremely  optimistic  plans.  Thus,  testers  and  analysts  sign  up  to  try  to  meet  aggressive 
schedules.  The  respondents  concluded  that  these  optimistic  plans  and  schedules  were 
unrealistic  and  based  on  meeting  the  schedule,  not  on  historical  test  information  or  test 
experience. 

b.  Findings  Across  Agencies 

Every  agency  identified  the  acquisition  process  as  a  problem.  The  PM 
representatives  focused  on  the  fear  of  losing  funding  and  support  as  the  cause  of  the 
problem.  One  tester  indicated  the  acquisition  process  was  the  major  problem.  He  believed 
the  main  cause  was  the  cunent  incentive  system  that  rewarded  PMs  for  being  unrealistically 
optimistic.  Two  AMSAA  representatives  stated  that  the  acquisition  process  was  the  major 
problem  area  and  pointed  to  unrealistic  early  estimates  as  the  cause.  Two  Combat 
Developers  also  determined  the  process  was  the  major  problem.  One  believed  the  cause 
was  the  layers  of  bureaucracy  and  paperwork.  The  other  Combat  Developer  regarded 
inconsistencies  within  the  process  as  the  cause.  The  contractor  also  determined  the 
inconsistencies  as  the  cause  of  problems.  For  example:  the  military  is  told  buy  "off  the 
shelf  items,"  but  the  items  must  still  meet  rigid  requirements  and  standards  that  some 
"shelT  items  cannot  possibly  meet 
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c.  Recommendations 

The  assumption  for  most  of  the  respondents  was  that  the  acquisition  process 
will  not  or  cannot  be  changed  or  reformed  enough  to  impact  DT&E,  The  recommendations 
to  improve  acquisition  process  problems  were  based  on  that  assumption  and  include: 

•  PMs  should  push  for  early  involvement  of  all  participants  including  the  testers  and 
evaluators. 

•  PMs  should  hold  participative  and  conclusive  TIWGs  to  address  test  plans  and 
schedules. 

•  The  Combat  Developer  should  be  involved  early  and  play  a  definitive  role. 

•  The  analysts  should  utilize  more  efficient  methods  of  evaluation,  more  modeling  and 
simulation  and  accept  more  risk. 

•  Senior  decision  makers  should  find  a  way  to  reward  PMs  who  are  critical  and 
objective. 

The  acquisition  process  was  considered  a  major  problem  in  conducting 
DT&E.  The  acquisition  process  was  also  cited  as  the  cause  of  some  of  the  other  categories 
of  problems  identified  in  this  thesis.  Early  and  definitive  involvement  from  the  tester,  the 
analyst  and  the  Combat  Developer  were  common  recommendations  for  addressing  this 
problem. 

3.  Test  Culture 

a.  Findings  Across  Systems 

The  Test  Culture  category  had  the  third  most  responses  overall.  The 
representative  causes  noted  for  this  problem  included  1)  the  acquisition  process  itself,  2) 
lack  of  PMO  understanding  of  test  and  analysis  capabilities  and  constraints,  and  3)  the 
assumed  reputation  of  testers  and  analysts  as  wanting  to  oveitest.  The  respondents 
believed  the  acquisition  process  drove  PMs  to  focus  on  cost  and  schedule  and  regard 
DT&E  as  an  opportunity  to  make  up  time  and  money.  Interviewees  also  indicated  that  PM 
Offices  may  not  realize  what  the  testers  and  evaluators  can  or  cannot  do  within  the 
constraints  of  the  budget  and  the  schedule.  Therefore,  unless  the  PMO  involves  the  tester 
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and  analyst  early,  PM  offices  could  develop  unrealistic  test  plans.  Finally,  some  testers 
and  analysts  have  earned  poor  reputations  among  Program  Offices  by  conducting  tests  or 
pursuing  additional  data  that  a  appeared  to  add  no  value  to  the  process.  This  practice  has 
caused  increased  costs  and  affected  the  credibility  of  testers  and  analysts. 

b.  Findings  Across  Agencies 

The  majority  of  the  responses  for  this  category  were  represented  by  test 
agencies.  Four  of  the  seven  respondents  in  this  category  were  testers;  AMSAA  and  one 
contractor  were  also  represented.  The  PM  representatives  and  the  Combat  Developers  as 
agencies  did  not  respond  in  this  category.  The  testers,  three  of  whom  were  supervisors 
within  TECOM,  believed  that  test  culture  was  root  of  many  cost  and  schedule  problems. 
T!:ey  pointed  to  the  acquisition  process  as  the  cause  of  this  culture.  The  process 
encourages  the  PM  to  move  through  testing  quickly.  If  problems  occur  in  testing,  the 
testers  and  the  analysts  are  usually  the  presenters  of  the  bad  news.  Bad  test  news  can  mean 
rescheduling  tests,  may  bring  into  question  system  need  and  validity  from  outsiders,  and 
affect  other  cost  and  schedule  issues  for  the  PM.  Tv/o  AMSAA  representatives  had 
responses  that  fit  into  this  category.  Both  pointed  to  lack  of  funding  and  the  funding  cycle 
as  the  cause  of  the  problem.  The  current  funding  system  does  not  allow  the  PM  efficient 
long  term  planning  and  in  turn  does  not  allow  the  testers  and  analysts  to  execute  long  term 
planning.  One  contractor  representative  believed  that  the  acquisition  process  was  the  major 
cause  of  this  negative  approach  to  testing  experienced  in  many  PM  Offices.  He  said  that 
the  process  causes  PMs  to  focus  on  cost  and  schedule  and  regard  reducing  DT&E  reduction 
as  an  opportunity  to  make  up  for  schedule  and  cost  overruns. 

c .  Recommendations 

The  recommendations  presented  by  the  respondents  for  fixing,  mitigating  or 
preventing  problems  in  Test  Culture  include; 
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•  PMs  should  push  for  early  involvement  of  all  the  participants  including  the  testers  and 
evaluators. 

•  PMs  should  ensure  that  AMSAA,  the  tester  and  the  contractor  closely  coordinate  the 
test  effort. 

•  PMs  and  contractors  should  realize  the  value  tl  lat  DT&E  provides  to  development. 

•  Testers  should  educate  PMs  on  their  c^abilities  and  demonstrate  more  flexibility  in 
packaging  test  programs. 

•  Testers  should  become  more  familiar  with  the  systems  under  test. 

•  Combat  Developers  should  develop  and  stick  to  solid,  realistic  requirements. 

•  The  PM  must  make  the  testers  and  evaluators  part  of  the  team  and  not  the  bad  news 
messengers. 

Test  Culture  problems  were  generally  recognized  by  testers  and  analysts.  The 
causes  noted  for  this  problem  included  the  acquisition  process  itself,  lack  of  PMO 
understanding  of  test  and  analysis  capabilities  and  constraints,  and  the  assum^'d  reputation 
of  testers  and  analysts  as  wanting  to  overtest.  To  prevent  or  minimize  this  p.  jblem  most 
respondents  determined  that  PMs  should  make  the  test  community  (testers,  analysts)  part  of 
the  team  and  the  test  community  should  better  educate  PMs,  contractors  and  Combat 
Developers  of  their  respective  DT&E  capabilities  and  limitations. 

4.  Resources  Management 

a.  Findings  Across  Systems 

Resource  management  was  mentioned  as  a  problem  by  three  of  the  seven 
systems.  Respondents  indicated  that  the  causes  of  this  problem  included  short  term 
funding  and  limited  resources  (hardware  and  software)  for  DT&E.  A  system  entering 
DT&E  awaiting  funding  may  not  receive  the  resources  in  the  lead  time  needed  for  proper 
test  conduct.  Lack  of  funding  could  delay  test  setup,  delay  instrumentation/equipment 
checks,  cause  inconclusive  or  even  useless  early  test  results  and  reduce  needed  test  support 
personnel.  Short  term  funding  also  causes  PMs  to  desire  and  plan  for  perfect  success  in 
the  test  process.  Anything  other  than  perfect  success  could  impact  future  funds. 


Systems  under  development  are  often  constrained  by  limited  prototypes,  test 
models,  versions  of  softwiu’e,  and  other  components.  Other  required  events  of  a  system's 
development  could  cause  these  limited  resources  to  be  spread  across  the  country  and  not  at 
the  test  facility  in  time  for  proper  test  preparation.  The  lack  of  resources  at  the  right  place  at 
the  right  time  could  severely  affect  test  schedule,  the  test  conduct  or  the  evaluation  and 
reporting  process. 

I 

b.  Findings  Across  Agencies 

All  agencies  were  represented  in  this  category  except  the  testers.  The  PM 
representative  indicated  that  resources  management  was  the  major  problem  and  believed  the 
cause  was  the  lack  of  aggressive  management  by  the  PMO.  The  two  analysts  from 
AMSAA  believed  that  limited  funding  was  the  cause  for  this  problem.  '  nrime 
contractor  pointed  to  reduced  funding  and  compressed  schedule  as  the  cause  of  resource 
management  problems.  The  Combat  Developer  believed  the  problem  existed  because  of  the 
increasing  use  of  software  in  modem  systems.  Software  testing,  like  the  entiie  so^warc 
management  issue,  lacks  in  information,  experience  and  guidance  is  a  new 

environment  and  creates  many  problems  for  testing  as  well  as  other  functic. 

c .  Recommendations 

The  actions  recommended  by  the  respondents  to  resolve  or  at  least  minimize 
Resource  Management  problems  include: 

•  The  PM  should  require  intensive,  proactive  management  at  all  levels. 

•  PMs  should  plan  for  contingencies  and  not  assume  perfect  success  in  the  test  process. 

•  PMs  fund  testing  to  insure  test  resources  are  on  hand  when  needed  so  that  testers  can 
be  in  a  position  to  adjust  to  change. 

•  All  those  involved  in  the  DT&E  process  need  to  be  timely  with  their  input  into  the  test 
plan. 

•  Agencies  should  provide  solid,  realistic  estimates  of  resources  in  terms  of  both  time 
and  dollars. 
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Testers  should  become  more  familiar  with  the  systems  under  test  especially  software 
intensive  systems. 


Resources  Management  was  another  common  problem  area.  The  lack  of 
proper  resources  at  the  right  place,  at  the  right  time  could  severely  affect  the  test  and 
evaluation  of  a  system.  Short  term  funding  and  limited  resources  for  DT&E  were  noted  as 
the  causes  of  this  problem.  The  recommendations  that  addressed  this  problem  included: 
fund  testing  to  insure  test  resources  are  on  hand  when  needed;  tester  familiarity  with 
systems  under  test,  especially,  software  intensive  systems;  and  incorporating  more  realistic 
estimates  of  test  resources  required  by  all  agencies. 

I  ) 

5.  Change  In  Requirements 

a.  Findings  Across  Systems 

Change  in  (technical)  Requirements  is  the  final  problem  area  in  which 
responses  were  categorized.  Four  of  those  interviewed  described  this  as  the  major  problem 
or  issue  in  conducting  DT&E.  These  four  responses  represented  four  different  systems. 
They  indicated  that  the  causes  of  this  problem  were  the  lack  of  coordination  and  or 
communication  between  agencies  and  the  lack  of  understanding  of  DT&E  process  among 
the  Combat  Developers.  Lack  of  communication  and  coordination  results  in  major 
documents  such  as  the  ORD,  the  TEMP,  and  the  contract,  not  matching  up  with 
requirements.  It  causes  difHculties  in  defining  test  requirements  and  makes  test  plans  and 
conduct  more  difficult  and  expensive  than  originally  estimated.  Combat  Developers  who 
may  not  be  familiar  with  the  test  process  may  not  realize  the  impact  that  a  requirement 
change  could  have  on  the  test  and  evaluation  process. 

b.  Findings  Across  Agencies 

Across  agencies  there  was  one  response  for  each  of  the  agencies  except  from 
the  combat  developers.  None  of  the  five  Combat  Developers  or  user  representatives 
determined  that  change  in  requirements  was  the  biggest  problem  in  DT&E.  This  is  notable 
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since  these  agencies  are  most  likely  to  generate  changes  that  impact  the  PM,  the  testers  and 
the  evaluators.  The  PM  representative  believed  the  lack  of  good  communication  and 
cooperation  among  agencies  was  the  cause  of  the  problem.  The  tester  and  the  AMSAA 
representative  believed  the  lack  of  coordination  and  lack  of  understanding  of  the  T&E 
process  on  the  part  of  the  user  representative  was  the  cause.  The  contractor  representative 
pointed  to  both  coordination  and  communication  as  the  basis  for  this  problem, 
c.  Recommendations 

The  recommendations  for  reducing  requirement  changes  include: 

•  The  PM  should  insist  that  a  solid,  stable  and  realistic  ORD  be  maintained. 

•  PMs  should  hold  participative  and  conclusive  TIWGs. 

•  The  PM  should  establish  a  better  working  relationship  among  the  agencies  in  defining 
test  requirements. 

•  The  Combat  Developer  should  appreciate  the  impact  that  changing  requirements  has  on 
the  system  and  the  test  process. 

•  The  PM  should  ensure  that  the  major  documents,  to  include  the  contract,  are  closely 
coordinated. 

Change  in  Requirements  was  the  fifth  category  of  signiHcant  problems.  The 
respondents  cited  the  lack  of  coordination  and/or  communication  between  agencies  and  the 
lack  of  understanding  of  DT&E  process  among  the  Combat  Developers  as  the  causes  of 
this  problem.  Overall,  the  agencies  believed  that  strong  PM  management  of  people, 
resources  and  critical  documents  was  the  best  way  to  prevent  problems  associated  with 
change  in  requirements. 

C.  OTHER  OBSERVATIONS 

Analysis  was  also  conducted  to  determine  if  the  type  of  system  or  its  development 
strategy  influenced  the  problems  that  occurred.  This  analysis  used  information  gathered 
from  the  seven  systems  and  did  not  include  responses  from  the  inter\'iews  conducted  with 
supervisors  at  AMSAA  and  TECOM.  Analysis  by  type  of  system  (one  aircraft,  three 
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ground  vehicles,  one  antitank  weapon  and  two  communication/data  and  information 
systems)  revealed  little  relationship  between  the  categories  of  problems  and  the  type  of 
system.  This  could  imply  that  the  problem  areas  identiHed  in  this  thesis  are  generally 
applicable  to  Army  systems  regardless  of  type  of  end  item.  It  also  indicates  that  the 
recommendations  that  address  these  problems  may  be  t^plicable  to  various  Army  systems. 

Responses  were  then  analyzed  by  the  development  strategy  of  the  systems  against  the 
categories  of  problems.  The  development  strategies  for  the  systems  reviewed  were:  full 
development,  major  upgrade  to  an  existing  system  and  Non-Developmental  Item  .  The 
results  are  depicted  in  TABLE  II. 


TABLE  II 

DEVELOPMENT  VERSUS  PROBLEMS 


PROBLEM 

Development 

Schedule 

Acquisition 

Process 

Test 

Culture 

BSS^SiSi 

Change  in 
Requirements 

XX 

XX 

XX 

X 

XX 

Upgrade 

X 

* 

XX 

X 

NDl 

X 

X 

X 

X 

X-  A  single  agency  whose  responses  applied  to  that  particular  category. 


TABLE  D  appears  to  indicate  that  overall,  the  type  of  developntent  strategy  may  have 
minimal  influence  on  the  DT&E  problems  experienced  by  a  system.  However,  there  are 
two  areas  where  a  relationship  may  exist  (shaded):  1)  Schedule  problems  and  upgrades  in 
developoKnt  and  2)  the  acquisition  process  and  NDl  developments. 


1.  Schedule  and  Upgrades 

Two  systems,  both  representing  upgrades  in  development,  had  at  least  three  of  the 
agencies  for  their  respective  systems  indicated  that  schedule  was  the  major  problem  in 
conducting  DT&E.  This  indicated  that  a  relationship  may  exist  between  schedule  type 
problems  and  upgrade  type  developments.  The  cause  for  this  relationship  may  be  that 
"upgrades'*  are  often  seen  as  simply  integrating  new  components  and  subsystems.  DT&E 
schedules  are  developed  to  focus  on  the  upgrade.  Upgrades  however,  may  be  extensive 
and  incorporate  the  latest  technology  and  the  "simple  integration"  may  prove  more  difficult 
than  anticipated.  The  early  test  schedule  and  planning  for  the  upgrade  probably  does  not 
anticipate  the  impact  of  new  technologies  and  the  significance  of  the  integration  on  the  old 
system. 

2.  Acquisition  Process  and  NDl 

Problems  with  the  acquisition  process  and  NDI  developments  also  seem  to  be 
associated.  There  were  two  systems  in  the  NDl  spectrum  of  development.  Most  of  the 
respondents,  including  both  the  PM  representatives,  for  these  systems  indicated  that  the 
acquisition  process  was  the  major  problem  in  conducting  DT&E.  The  likely  cause  of  this 
relationship  is  that  NDl  developments  are  often  seen  by  senior  leaders  and  the  Congress  as 
a  kind  of  panacea  acquisition  model.  NDI  should  be  a  more  expedited  process,  to  include 
DT&E.  However,  the  acquisition  process,  as  previously  mentioned,  already  proliferates 
over  optimism  which  is  accentuated  for  NDl  developments.  This  aeates  an  environment  of 
extremely  high,  unrealistic  expectations  for  NDI  developments.  When  these  expectations 
are  not  realized  within  the  original  cost  and  schedule  estimates,  the  system  becomes  the 
target  of  scrutiny  and  question. 
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3.  Recommendations 

The  recommendations  that  address  these  two  relationships  include: 

•  PMs,  as  well  as  others,  should  avoid  underestimating  the  DT&E  process  for  NDl  and 
system  upgrades. 

*  Historical  test  and  analysis  information  and  data  firom  components  and  subsystems 
should  be  used  to  better  estimate  future  tests  costs  and  schedule. 

•  PMs  should  hold  participative  and  conclusive  TIWGs  to  address  test  plans  and 
schedules. 

♦  More  realistic  estimates  should  be  made  by  all  agencies  involved  with  less  optimism 
from  the  PM. 

*  PMs  should  push  for  early  involvement  of  all  the  participants  including  the  testers  and 
evaluators. 

Overall,  the  type  of  development  strategy  may  have  minimal  influence  on  the 
DT&E  problems  experienced  by  a  system.  However,  there  are  two  areas  where  a 
relationship  may  exist:  1)  Schedule  problems  and  upgrades  in  development,  and  2)  the 
acquisition  process  and  NDI  developments.  Both  relationships  may  be  the  result  of  the 
high  expectations  of  these  types  of  development  efforts.  Many  of  the  previous 
recommendations  for  schedule  problems  and  acquisition  problems  hold  true  for  these  two 
relationships.  In  addition  PMs,  as  well  as  others,  should  avoid  underestimating  the  DT&E 
process  for  NDI  and  system  upgrades  simply  because  they  are  "supposed  to  be  easier," 


V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

The  primary  purpose  of  this  thesis  was  to  explore  and  identify  recurring  problems  in 
developmental  testing  in  the  United  States  Army,  analyze  the  problems  and  make 
recommendations  to  prevent  and  or  minimize  these  problems.  As  a  result  of  this 
comparative  analysis,  it  appears  that  Developmental  Test  and  Evaluation  is  subject  to  many 
of  the  same  problems  that  occur  in  the  acquisition  process.  This  chapter  presents  the 
conclusions  and  recommendations  derived  from  the  analysis  of  the  previous  chapter. 

B.  GENERAL  CONCLUSIONS 

This  thesis  concludes  that  Hve  significant  problem  areas  exist  in  conducting 
Developmental  Test  and  Evaluation.  In  order  of  significance  these  problems  are:  1) 
Schedule  Problems,  2)  Problems  with  the  Acquisition  Process,  3)  Test  Culture  Problems, 
4)  Resources  Management  Problems  and  S)  Problems  with  Changing  Requirements.  The 
thesis  also  concludes  that  the  type  of  development  strategy  may  influence  which  of  these 
problems  is  most  prevalent.  Finally,  the  thesis  concludes  that  the  most  recognized  method 
to  alleviate  or  prevent  these  problems  is  for  the  PM  to  involve  the  tester,  the  analyst  and 
Combat  Developer  early  in  development. 

C.  SPECIFIC  CONCLUSIONS 

1 .  Schedule 

Schedule  problems  were  the  most  common  and  the  most  significant  problems  in 
conducting  DT&E.  Schedule  problems  are  caused  by  the  acquisition  process  which 
encourages  over  optimism,  unrealistic  schedule  estimates  and  emphasizes  completing  the 
test  on  schedule  over  conducting  the  test  according  to  plan.  The  process  may  cause  the  P>^ 


and  his  staff  to  develop  early  estimates  without  considering  historical  test  information  or  the 
experience  of  the  tester  or  analyst 

2 .  Acquisition  Process 

The  acquisition  process  itself  presents  a  significant  problem  to  conducting  DT&E 
as  well  as  being  a  cause  of  other  related  problems.  Nearly  every  agency  addressed  the 
acquisition  process  as  a  major  problem.  The  causes  for  this  problem  included  the  funding 
process  and  PM  over  optimism.  The  funding  process  rewards  PMs  for  being  on  schedule, 
under  budget  and  meeting  the  criteria  of  the  next  milestone,  but  not  for  being  critical  and 
objective  about  their  system  and  not  for  taking  a  user  perspective.  Over  optimism  by  the 

PM  in  his  planning  and  scheduling,  forces  other  agencies  in  turn  to  sign  up  to  unrealistic 

\ 

plan's  that  are  based  on  meeting  an  aggressive  schedule  not  based  on  the  sy.stem's  readiness 
for  testing. 

3.  Test  Culture 

This  thesis  concluded  that  a  negative  test  culture  exists  and  this  culture  was  the 
basis  of  many  DT&E  problems.  PMs,  their  staffs,  and  sometimes  contractors  have  a 
negative  attitude  toward  testing,  testers  and  analysts.  The  representative  causes  noted  for 
this  problem  included:  1)  the  acquisition  process  itself,  2)  lack  of  PMO  understanding  of 
test  and  analysis  capabilities  and  constraints,  and  3)  the  assumed  reputation  that  testers  and 
analysts  require  excessive  testing. 

The  acquisition  process  drives  PMs  to  focus  on  cost  and  schedule  and  regard 
DT&E  as  an  opportunity  to  make  up  time  and  money.  PM  Offices  may  not  realize  what  the 
testers  and  evaluators  can  or  cannot  do  for  the  PM  unless  the  PMO  involves  the  tester  and 
the  analyst  early.  Some  testers  and  analysts  have  earned  poor  reputations  among  Program 
Offices  by  conducting  tests  fnat  appeared  to  add  no  value  to  the  process. 
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4.  Resources  Management 

Resources  management  of  critical  test  assets  was  another  major  problem  in 
conducting  DT&E.  The  causes  of  this  problem  included  short  term  funding  and  limited 
resources  (hardware  and  software).  A  system  entering  DT&E  without  adequate  test 
funding  may  not  receive  the  resources  in  the  lead  time  needed  for  proper  test  conduct.  Lack 
of  funding  could  delay  test  setup,  delay  instnimentation/equipment  checks,  and  reduce 
needed  test  support  personnel.  Short  term  funding  also  caused  PMs  to  desire  and  plan  for 
perfect  success  in  the  test  process.  Systems  under  development  are  often  constrained  by 
limited  prototypes,  test  models,  versions  of  software,  and  may  be  spread  across  the 
country.  The  lack  of  resources  can  severely  limit  effective  testing  evaluation  and  reporting. 

5.  Change  in  Requirements 

Changes  in  requirements  were  a  major  problem  for  DT&E.  The  causes  of  this 
problem  were  the  lack  of  coordination  and/or  communication  between  agencies  and  the  lack 
of  understanding  of  DT&E  process  among  the  Combat  Developers.  Lack  of 
communication  and  coordination  resulted  in  documents  such  as  the  ORD,  the  TEMP,  and 
the  contract,  not  matching  in  terms  of  requirements.  It  caused  difficulties  in  defining  test 
requirements  and  made  test  plans  and  test  conduct  more  difficult  and  expensive  than 
originally  estimated.  Combat  Developers,  the  agency  where  most  changes  come  from,  may 
not  be  familiar  with  the  test  process  and  may  not  realize  the  impact  that  a  requirement 
change  has  on  the  test  and  evaluation  process. 

6.  Other  Conclusions 

a.  Developmental  Strategy 

This  thesis  also  concluded  that  a  system's  development  strategy  may  be 
related  to  the  type  of  problems  a  system  encounters.  Two  particular  areas  which  reveal 
strong  relationships  were  1)  Schedule  problems  and  upgrades  in  development  and  2)  the 
acquisition  process  and  NDI  developments.  The  main  cause  for  both  these  relationships 
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was  that  these  types  of  developments  tend  to  promote  very  high  expectations  among  PMs, 
senior  decision  makers  and  other  agencies.  It  has  often  been  anticipated  that  there  should 
be  minimal  problems  in  the  DT&E  of  such  developments  (although  the  contrary  is  more 
likely).  Therefore,  when  cost  and  schedule  overruns  occur  prior  or  during  DT&E,  the 
senior  decision  makers,  other  agencies  and  even  the  Congress  scrutinize  and  reassess  the 
system. 

b.  Early  Involvement 

Early  involvement  of  the  tester,  the  analyst  and  the  Combat  Developer  is 
critical  to  minimizing  and  or  preventing  DT&E  problems.  Having  the  PM  bring  these 
agencies  in  early  to  help  estimate,  plan,  and  coordinate  the  test  effort  was  the  most  common 
recommendation  made.  This  recommendation  was  observed  across  systems,  agencies,  and 
all  categories  of  problems. 

D.  RECOMMENDATIONS 

To  improve  Developmental  Test  and  Evaluation,  Program  Managers,  testers,  analyst, 
Combat  Developer?  and  contractors  should  review  and  address  the  DT&E  problems 
identified  in  this  thesis.  Specifically  they  should  be  prepared  to  address  and  account  for 
problems  involving:  1)  Schedule,  2)  the  Acquisition  Process,  3)  Test  Culture  4)  Resources 
Management  and  5)  Changing  Requirements,  in  that  order. 

1 .  General  Recommendations 

The  PM  should  bring  in  all  agencies  for  early  planning,  especially  the  tester,  the 
analyst,  and  the  Combat  Developer.  The  PM’s  DT&E  effort  should  concentrate  on  realistic 
estimates  of  test  cost  and  schedule,  make  the  test  community  part  of  the  team,  aggressively 
manage  test  resources,  and  foster  a  working  relationship  between  agencies  that  emphasizes 
cooperation  and  communication.  The  following  is  a  list  of  general  recommendations  to 
prevent  or  minimize  the  problems  identified  in  this  thesis: 
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•  PMs  should  push  for  early  involvement  of  all  the  participants  including  the  testers, 
analysts  and  Combat  Developers. 

•  The  PM  must  make  the  testers  and  analysts  part  of  the  team  and  not  the  bad  news 
messengers. 

•  Agencies  should  provide  solid,  realistic  estimates  of  resources  in  terms  of  both  time 
and  dollars. 

•  Testers  should  educate  PMs  on  their  capabilities  and  demonstrate  more  flexibility  in 
packaging  test  programs. 

•  Combat  Developers  should  develop  and  stick  to  solid,  realistic  requirements. 

2.  Specific  Recommendations 

The  following  recommendations  are  made  to  address  each  of  the  specific 
categories  noted  in  the  thesis. 

a.  Schedule 

•  Starting  with  the  PM  and  his  staff,  more  realistic  schedule  estimates  should  be  made 
by  all  agencies  involved 

•  PMs  should  hold  participative  and  conclusive  TIWGs  to  address  test  plans  and 
schedules. 

•  Historical  information  and  data  from  previous  tests  should  be  used  to  better  estimate 
future  test  schedules. 

h.  Acquisition  Process 

•  The  analysts  should  not  promote  excessive  testing  and  should  integrate  other  and  more 
e^icient  methods  of  evaluation  including  modeling  and  simulation. 

•  Senior  decision  makers  should  reward  PMs  who  are  realistic  and  objective  about  the 
development  of  their  system. 

c .  Test  Culture 

•  PMs  should  ensure  that  AMSAA,  the  tester  and  the  contractor  closely  coordinate  the 
test  effort 

•  PMs  and  contractors  should  realize  the  value  diat  DT&E  provides  to  their  development 
effort. 

•  Testers  should  educate  PMs  on  their  capabilities  and  demonstrate  more  flexibility  in 
packaging  test  programs. 
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•  The  PM  must  make  the  testers  and  analysts  part  of  the  team  and  not  the  bad  news 
messengers. 

•  Testers  and  analysts  should  become  more  familiar  with  the  systems  under  test  to  better 
understand,  "What  to  test?" 

d.  Resources  Management 

•  The  PM  should  require  intensive,  proactive  management  at  all  levels. 

•  PMs  should  plan  for  contingencies  and  not  assume  perfect  success  in  the  test  process. 

•  PMs  should  fund  testing  to  insure  test  resources  are  available  when  needed  for  proper 
test  conduct 

•  Testers  should  become  more  familiar  >vith  the  systems  under  test  especially  software 
intensive  systems. 

e.  Change  in  Requirements 

•  The  PM  should  insist  that  a  solid,  stable  and  realistic  ORD  be  maintained. 

•  The  PM  should  establish  a  better  working  relationship  among  the  agencies  in  defining 
test  requirements. 

•  The  Combat  Developer  should  appreciate  the  impact  that  changing  requirements  has  on 
the  system  and  the  test  process. 

•  The  PM  should  ensure  that  the  major  documents,  to  include  the  contract,  are  closely 
coordinated. 

/.  Other  Recommendations 

The  following  recommendations  focus  on  upgrade  and  NDI  type 
developments. 

•  PMs,  as  well  as  others,  5should  avoid  underestimating  the  DT&E  process  for  NDI  and 
system  upgrades. 

•  Historical  test  data  and  analysis  information  from  components  and  subsystems  should 
be  used  to  better  estimate  future  tests,  costs,  and  schedule. 

•  PMs  should  h^id  early  TIWGs  to  address  unique  testing  requirements,  plans  and 
schedules. 


E.  AREAS  FOR  FURTHER  RESEARCH 

The  following  areas  should  be  investigated  for  potential  benefit  to  the  DOD: 

•  Developmental  Strategy  and  Test  Problems  -  One  of  the  findings  of  this  thesis 
was  that  the  type  of  development  strategy  influenced  the  number  of  test  problems  a 
system  encountered.  Further  research  on  the  effect  of  development  strategy  on  test 
programs  could  provide  insight  for  better  tailoring  of  programs. 

•  Test  Culture  -  Developmental  Test  and  Evaluation  could  be  significantly  improved  if 
the  relationship  between  the  test  community  (testers  and  analysts)  and  the  PM  Office 
were  improved.  Further  research  into  the  causes  of  this  adversarial  relationship  with 
emphasis  on  preventing  or  minimizing  its  impact  on  testing  could  provide  valuable 
information  to  the  DOD. 

•  PM  Incentives  nnd  the  Acquisition  Process  -  Researching  the  feasibility  of 
incorporating  an  incentive  system  for  PMs  which  encouraged  them  to  be  "event  driven 
and  user  oriented"  should  be  conducted.  A  meaningful  and  quantifiable  rating  and 
evaluation  system  for  PMs  that  stressed  good  management  techniques  vice  political 
maneuvering  should  be  developed. 
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APPENDIX  A  (PROGRAM  MANAGEMENT  OFFICE  QUESTIONS) 


Name:  Location: 

Title  or  Job  Date: 

Thank  you  for  taking  the  time  to  meetAalk  with  me.  I  am  conducting  this  research  for  my 
master’s  thesis  at  the  Naval  Postgraduate  School.  The  focus  of  my  diesis  is  to  identify 
problems  in  developmental  testing  from  the  perspective  of  the  various  agencies 
involved  in  DT&E,  In  my  final  product  I  hope  to  nru^e  some  recommendations  to  correct 
the  problems  or  at  least  minimize  the  impact  of  these  problems  for  future  programs.  I  am 
examining  a  number  of  systems  and  interviewing  the  key  players  in  die  developmental  test 
process  of  those  systems  as  well  as  some  others.  My  questions  to  you  will  primarily 
address  your  interface  with  these  other  agencies,  what  you  perceive  as  the  major  problem  in 
DT&E  and  how  can  you  or  the  other  agencies  solve  or  alleviate  the  problem?  I  would  also 
welcome  any  additional  comments  that  you  have  about  any  aspect  of  the  developmental  test 
process.  The  information  1  collect  will  be  confidential  if  you  request.  I  will  consolidate 
and  summarize  all  interview  data  so  that  your  name  will  not  be  identified  in  any  way. 


PROGRAM.  MANAGERS 


1 .  How  do  you  interface  with  these  agencies: 


-Test  facility 


-AMSAA 


-The  contractor 


-The  Combat  Developer  (user  representative) 

2.  is  this  sufficient? 

3 .  What  do  you  consider  the  mt^t  significant  problem  or  issue  in 
entering  a  Developmental  Test? 

4 .  Does  this  problem  effect  the  cost  and  or  scheduling  of  the  DT? 
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5 .  What  rules,  regulations  or  policies  exist  that  try  to  alleviate  this  problem?  What 
rules,  regulations  or  policies  only  feed  the  problem? 

6 .  Does  the  problem  impact  future  testing  -  DT  or  OT?  How? 

7 .  What  could  you  the  PM  do  to  deal  with  this  problem? 

8 .  What  could  be  done  by  the  Tester  to  deal  with  this  problem? 

-Are  they  aware  of  the  problem? 

9 .  What  could  be  done  by  the  Contractor  to  deal  with  this  problem? 

-Are  they  aware  of  the  problem? 

10.  What  could  be  done  by  the  Combat  Developer  to  deal  with  this  problem? 

-Are  they  aware  of  the  problem? 

1 1 .  What  could  be  done  by  Agencies  like  AMS  A  A  to  deal  with  this  problem? 

-Are  they  aware  of  the  problem? 

1 2.  What  could  be  done  by  Legislative  action  to  deal  with  this  problem? 

1 3 .  What  do  you  consider  as  other  key  problems  or  issues  in  Developmental  Testing? 

1 4.  What  impact  do  these  problems  have  on  the  cost  and  schedule  of  the  test? 

15.  What  could  be  done  by  who  and  how  to  deal  with  these  problems? 
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APPENDIX  B  (LIST  OF  ACRONYMS) 


AAE 

Anny  Acquisition  Executive 

ACAT 

Acquisition  Categoiy 

ADM 

Acquisition  Decision  Memorandum 

AMC 

Army  Materiel  Command 

AMSAA 

Army  Materiel  Systems  Analysis  Activity 

ARL 

Army  Research  Laboratory 

ASA  (RDA) 

Assistant  Secretary  Of  The  Army  (Research,  Development  &. 
Acquisition) 

ASD 

Assistant  Secretary  of  Defense 

ATCCS 

Army  Tactical  Command  and  Control  System 

CE 

Concept  Exploration  (Phase) 

CLU 

Command  Launch  Unit 

OTV 

Commander's  Independent  Thermal  Viewer 

COEA 

Cost  and  Operational  Effectiveness  Analysis  (Report) 

COMSEC 

Communications  Security 

DA 

Department  of  the  Army 

DAB 

Defense  Acquisition  Board 

DAE 

Defense  Acquisition  Executive 

D&V 

Demonstration  and  Validation 

DDT&E 

Director,  Defense  Test  and  Evaluation 

DEMVAL 

Demonstration  and  Validation  Phase 

IXX> 

Department  of  Defense 

DODD 

Department  of  Defense  Directive 

DODI 

Department  of  Defense  Instruction 

DOT&E 

Director  Operational  Test  and  Evaluation 

ET 

Development  Test 

DTE 

Director,  Test  and  Evaluation 

DT&E 

Development  Test  and  Evaluation 

DUSA  (OR) 

Deputy  Under  Secretary  Army  (Operations  Research) 

ECP 

Engineering  Change  Proposal 

ECCM 

Electronic  Counter-Countermeasures 

ec:m 

Electronic  Countermeasures 

EMD 

Engineering  and  Manufacturing  Development 

ESM 

Electronic  Support  Measure 

EW 

Electronic  Warfare 

FAT 

First  Article  Test 

FLIR 

Forward-Looking  Infrared 

FMTV 

Family  of  Medium  Tactical  Vehicles 

FOT&E 

FoUow-on  (^rational  Test  and  Evaluation 

GAO 

General  Accounting  Office 
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